delira - Deep Learning In RAdiology¶

Getting started¶

Backends¶

Before installing delira, you have to choose a suitable backend. delira handles backends as optional dependencies and tries to escape all uses of a not-installed backend.

The currently supported backends are:

torch (recommended, since it is the most tested backend): Suffix torch

Note

delira supports mixed-precision training via apex, but apex must be installed separately
tf (very experimental): Suffix tensorflow

Note

the tensorflow backend is still very experimental and may be unstable.
None: No Suffix
All (installs all registered backends and their dependencies; not recommended, since this will install many large packages): Suffix full

Note

Depending on the backend, some functionalities may not be available for you. If you want to ensure, you can use each functionality, please use the full option, since it installs all backends

Note

If you want to add a backend like CNTK, Chainer, MXNET or something similar, please open an issue for that and we will guide you during that process (don’t worry, it is not much effort at all).

Installation¶

Backend	Bin ary Ins tal lat ion	Source Installation	Notes
None	``p ip ins tal l d eli ra` `	`pip install git+ https://github.com /justusschock/deli ra.git`	Training not possible if backend is not installed separately
torch _	``p ip ins tal l d eli ra[ tor ch] ``	``git clone https: //github.com/justu sschock/delira.git && cd delira && p ip install .[torch ]``	`delira` with `torch` backend supports mixed-precision training via NVIDIA/apex (must be installed separately).
`tensor flow`_	``p ip ins tal l d eli ra[ ten sor flo w]` `	``git clone https: //github.com/justu sschock/delira.git && cd delira && p ip install .[tenso rflow]``	the `tensorflow` backend is still very experimental and lacks some features
Full	``p ip ins tal l d eli ra[ ful l]` `	``git clone https: //github.com/justu sschock/delira.git && cd delira && p ip install .[full] ``	All backends will be installed.

Delira Introduction¶

Last updated: 09.05.2019

Authors: Justus Schock, Christoph Haarburger

Loading Data¶

To train your network you first need to load your training data (and probably also your validation data). This chapter will therefore deal with delira’s capabilities to load your data (and apply some augmentation).

The Dataset¶

There are mainly two ways to load your data: Lazy or non-lazy. Loading in a lazy way means that you load the data just in time and keep the used memory to a bare minimum. This has, however, the disadvantage that your loading function could be a bottleneck since all postponed operations may have to wait until the needed data samples are loaded. In a no-lazy way, one would preload all data to RAM before starting any other operations. This has the advantage that there cannot be a loading bottleneck during latter operations. This advantage comes at cost of a higher memory usage and a (possibly) huge latency at the beginning of each experiment. Both ways to load your data are implemented in delira and they are named BaseLazyDatasetand BaseCacheDataset. In the following steps you will only see the BaseLazyDataset since exchanging them is trivial. All Datasets (including the ones you might want to create yourself later) must be derived of delira.data_loading.AbstractDataset to ensure a minimum common API.

The dataset’s __init__ has the following signature:

def __init__(self, data_path, load_fn, **load_kwargs):

This means, you have to pass the path to the directory containing your data (data_path), a function to load a single sample of your data (load_fn). To get a single sample of your dataset after creating it, you can index it like this: dataset[0]. Additionally you can iterate over your dataset just like over any other python iterator via

for sample in dataset:
    # do your stuff here

or enumerate it via

for idx, sample in enumerate(dataset):
    # do your stuff here

.

The missing argument **load_kwargs accepts an arbitrary amount of additional keyword arguments which are directly passed to your loading function.

An example of how loading your data may look like is given below:

from delira.data_loading import BaseLazyDataset, default_load_fn_2d
dataset_train = BaseLazyDataset("/images/datasets/external/mnist/train",
                                default_load_fn_2d, img_shape=(224, 224))

In this case all data lying in /images/datasets/external/mnist/train is loaded by default_load_fn_2d. The files containing the data must be PNG-files, while the groundtruth is defined in TXT-files. The default_load_fn_2d needs the additional argument img_shape which is passed as keyword argument via **load_kwargs.

Note: for reproducability we decided to use some wrapped PyTorch datasets for this introduction.

Now, let’s just initialize our trainset:

from delira.data_loading import TorchvisionClassificationDataset
dataset_train = TorchvisionClassificationDataset("mnist", train=True,
                                                 img_shape=(224, 224))

Getting a single sample of your dataset with dataset_train[0] will produce:

dataset_train[0]

which means, that our data is stored in a dictionary containing the keys data and label, each of them holding the corresponding numpy arrays. The dataloading works on numpy purely and is thus backend agnostic. It does not matter in which format or with which library you load/preprocess your data, but at the end it must be converted to numpy arrays For validation purposes another dataset could be created with the test data like this:

dataset_val = TorchvisionClassificationDataset("mnist", train=False,
                                               img_shape=(224, 224))

The Dataloader¶

The Dataloader wraps your dataset to privode the ability to load whole batches with an abstract interface. To create a dataloader, one would have to pass the following arguments to it’s __init__: the previously created dataset.Additionally, it is possible to pass the batch_size defining the number of samples per batch, the total number of batches (num_batches), which will be the number of samples in your dataset devided by the batchsize per default, a random seedfor always getting the same behaviour of random number generators and a `sampler <>`__ defining your sampling strategy. This would create a dataloader for your dataset_train:

from delira.data_loading import BaseDataLoader

batch_size = 32

loader_train = BaseDataLoader(dataset_train, batch_size)

Since the batch_size has been set to 32, the loader will load 32 samples as one batch.

Even though it would be possible to train your network with an instance of BaseDataLoader, malira also offers a different approach that covers multithreaded data loading and augmentation:

The Datamanager¶

The data manager is implemented as delira.data_loading.BaseDataManager and wraps a DataLoader. It also encapsulates augmentations. Having a view on the BaseDataManager’s signature, it becomes obvious that it accepts the same arguments as the `DataLoader <#The-Dataloader>`__. You can either pass a dataset or a combination of path, dataset class and load function. Additionally, you can pass a custom dataloder class if necessary and a sampler class to choose a sampling algorithm.

The parameter transforms accepts augmentation transformations as implemented in batchgenerators. Augmentation is applied on the fly using n_process_augmentation threads.

All in all the DataManager is the recommended way to generate batches from your dataset.

The following example shows how to create a data manager instance:

from delira.data_loading import BaseDataManager
from batchgenerators.transforms.abstract_transforms import Compose
from batchgenerators.transforms.sample_normalization_transforms import MeanStdNormalizationTransform

batchsize = 64
transforms = Compose([MeanStdNormalizationTransform(mean=1*[0], std=1*[1])])

data_manager_train = BaseDataManager(dataset_train,  # dataset to use
                                    batchsize,  # batchsize
                                    n_process_augmentation=1,  # number of augmentation processes
                                    transforms=transforms)  # augmentation transforms

The approach to initialize a DataManager from a datapath takes more arguments since, in opposite to initializaton from dataset, it needs all the arguments which are necessary to internally create a dataset.

Since we want to validate our model we have to create a second manager containing our dataset_val:

data_manager_val = BaseDataManager(dataset_val,
                                    batchsize,
                                    n_process_augmentation=1,
                                    transforms=transforms)

That’s it - we just finished loading our data!

Iterating over a DataManager is possible in simple loops:

from tqdm.auto import tqdm # utility for progress bars

# create actual batch generator from DataManager
batchgen = data_manager_val.get_batchgen()

for data in tqdm(batchgen):
    pass # here you can access the data of the current batch

Sampler¶

In previous section samplers have been already mentioned but not yet explained. A sampler implements an algorithm how a batch should be assembled from single samples in a dataset. delira provides the following sampler classes in it’s subpackage delira.data_loading.sampler:

AbstractSampler
SequentialSampler
PrevalenceSequentialSampler
RandomSampler
PrevalenceRandomSampler
WeightedRandomSampler
LambdaSampler

The AbstractSampler implements no sampling algorithm but defines a sampling API and thus all custom samplers must inherit from this class. The Sequential sampler builds batches by just iterating over the samples’ indices in a sequential way. Following this, the RandomSampler builds batches by randomly drawing the samples’ indices with replacement. If the class each sample belongs to is known for each sample at the beginning, the PrevalenceSequentialSampler and the PrevalenceRandomSampler perform a per-class sequential or random sampling and building each batch with the exactly same number of samples from each class. The WeightedRandomSampleraccepts custom weights to give specific samples a higher probability during random sampling than others.

The LambdaSampler is a wrapper for a custom sampling function, which can be passed to the wrapper during it’s initialization, to ensure API conformity.

It can be passed to the DataLoader or DataManager as class (argument sampler_cls) or as instance (argument sampler).

Models¶

Since the purpose of this framework is to use machine learning algorithms, there has to be a way to define them. Defining models is straight forward. delira provides a class delira.models.AbstractNetwork. All models must inherit from this class.

To inherit this class four functions must be implemented in the subclass:

__init__
closure
prepare_batch
__call__

`init`¶

The __init__function is a classes constructor. In our case it builds the entire model (maybe using some helper functions). If writing your own custom model, you have to override this method.

Note: If you want the best experience for saving your model and completely recreating it during the loading process you need to take care of a few things: * if using torchvision.models to build your model, always import it with from torchvision import models as t_models * register all arguments in your custom __init__ in the abstract class. A init_prototype could look like this:

def __init__(self, in_channels: int, n_outputs: int, **kwargs):
    """

    Parameters
    ----------
    in_channels: int
        number of input_channels
    n_outputs: int
        number of outputs (usually same as number of classes)
    """
    # register params by passing them as kwargs to parent class __init__
    # only params registered like this will be saved!
    super().__init__(in_channels=in_channels,
                     n_outputs=n_outputs,
                     **kwargs)

`closure`¶

The closurefunction defines one batch iteration to train the network. This function is needed for the framework to provide a generic trainer function which works with all kind of networks and loss functions.

The closure function must implement all steps from forwarding, over loss calculation, metric calculation, logging (for which delira.logging_handlers provides some extensions for pythons logging module), and the actual backpropagation.

It is called with an empty optimizer-dict to evaluate and should thus work with optional optimizers.

`prepare_batch`¶

The prepare_batchfunction defines the transformation from loaded data to match the networks input and output shape and pushes everything to the right device.

Abstract Networks for specific Backends¶

PyTorch¶

At the time of writing, PyTorch is the only backend which is supported, but other backends are planned. In PyTorch every network should be implemented as a subclass of torch.nn.Module, which also provides a __call__ method.

This results in sloghtly different requirements for PyTorch networks: instead of implementing a __call__ method, we simply call the torch.nn.Module.__call__ and therefore have to implement the forward method, which defines the module’s behaviour and is internally called by torch.nn.Module.__call__ (among other stuff). To give a default behaviour suiting most cases and not have to care about internals, delira provides the AbstractPyTorchNetwork which is a more specific case of the AbstractNetwork for PyTorch modules.

`forward`¶

The forward function defines what has to be done to forward your input through your network and must return a dictionary. Assuming your network has three convolutional layers stored in self.conv1, self.conv2 and self.conv3 and a ReLU stored in self.relu, a simple forward function could look like this:

def forward(self, input_batch: torch.Tensor):
    out_1 = self.relu(self.conv1(input_batch))
    out_2 = self.relu(self.conv2(out_1))
    out_3 = self.conv3(out2)

    return {"pred": out_3}

`prepare_batch`¶

The default prepare_batch function for PyTorch networks looks like this:

@staticmethod
def prepare_batch(batch: dict, input_device, output_device):
    """
    Helper Function to prepare Network Inputs and Labels (convert them to
    correct type and shape and push them to correct devices)

    Parameters
    ----------
    batch : dict
        dictionary containing all the data
    input_device : torch.device
        device for network inputs
    output_device : torch.device
        device for network outputs

    Returns
    -------
    dict
        dictionary containing data in correct type and shape and on correct
        device

    """
    return_dict = {"data": torch.from_numpy(batch.pop("data")).to(
        input_device)}

    for key, vals in batch.items():
        return_dict[key] = torch.from_numpy(vals).to(output_device)

    return return_dict

and can be customized by subclassing the AbstractPyTorchNetwork.

`closure example`¶

A simple closure function for a PyTorch module could look like this:

    @staticmethod
    def closure(model: AbstractPyTorchNetwork, data_dict: dict,
                optimizers: dict, criterions={}, metrics={},
                fold=0, **kwargs):
        """
        closure method to do a single backpropagation step

        Parameters
        ----------
        model : :class:`ClassificationNetworkBasePyTorch`
            trainable model
        data_dict : dict
            dictionary containing the data
        optimizers : dict
            dictionary of optimizers to optimize model's parameters
        criterions : dict
            dict holding the criterions to calculate errors
            (gradients from different criterions will be accumulated)
        metrics : dict
            dict holding the metrics to calculate
        fold : int
            Current Fold in Crossvalidation (default: 0)
        **kwargs:
            additional keyword arguments

        Returns
        -------
        dict
            Metric values (with same keys as input dict metrics)
        dict
            Loss values (with same keys as input dict criterions)
        list
            Arbitrary number of predictions as torch.Tensor

        Raises
        ------
        AssertionError
            if optimizers or criterions are empty or the optimizers are not
            specified

        """

        assert (optimizers and criterions) or not optimizers, \
            "Criterion dict cannot be emtpy, if optimizers are passed"

        loss_vals = {}
        metric_vals = {}
        total_loss = 0

        # choose suitable context manager:
        if optimizers:
            context_man = torch.enable_grad

        else:
            context_man = torch.no_grad

        with context_man():

            inputs = data_dict.pop("data")
            # obtain outputs from network
            preds = model(inputs)["pred"]

            if data_dict:

                for key, crit_fn in criterions.items():
                    _loss_val = crit_fn(preds, *data_dict.values())
                    loss_vals[key] = _loss_val.detach()
                    total_loss += _loss_val

                with torch.no_grad():
                    for key, metric_fn in metrics.items():
                        metric_vals[key] = metric_fn(
                            preds, *data_dict.values())

        if optimizers:
            optimizers['default'].zero_grad()
            total_loss.backward()
            optimizers['default'].step()

        else:

            # add prefix "val" in validation mode
            eval_loss_vals, eval_metrics_vals = {}, {}
            for key in loss_vals.keys():
                eval_loss_vals["val_" + str(key)] = loss_vals[key]

            for key in metric_vals:
                eval_metrics_vals["val_" + str(key)] = metric_vals[key]

            loss_vals = eval_loss_vals
            metric_vals = eval_metrics_vals

        for key, val in {**metric_vals, **loss_vals}.items():
            logging.info({"value": {"value": val.item(), "name": key,
                                    "env_appendix": "_%02d" % fold
                                    }})

        logging.info({'image_grid': {"images": inputs, "name": "input_images",
                                     "env_appendix": "_%02d" % fold}})

        return metric_vals, loss_vals, preds

**Note:** This closure is taken from the
``delira.models.classification.ClassificationNetworkBasePyTorch``

Other examples¶

In delira.models you can find exemplaric implementations of generative adversarial networks, classification and regression approaches or segmentation networks.

Training¶

Parameters¶

Training-parameters (often called hyperparameters) can be defined in the delira.training.Parameters class.

The class accepts the parameters batch_size and num_epochs to define the batchsize and the number of epochs to train, the parameters optimizer_cls and optimizer_params to create an optimizer or training, the parameter criterions to specify the training criterions (whose gradients will be accumulated by defaut), the parameters lr_sched_cls and lr_sched_params to define the learning rate scheduling and the parameter metrics to specify evaluation metrics.

Additionally, it is possible to pass an aritrary number of keyword arguments to the class

It is good practice to create a Parameters object at the beginning and then use it for creating other objects which are needed for training, since you can use the classes attributes and changes in hyperparameters only have to be done once:

import torch
from delira.training import Parameters
from delira.data_loading import RandomSampler, SequentialSampler

params = Parameters(fixed_params={
    "model": {},
    "training": {
        "batch_size": 64, # batchsize to use
        "num_epochs": 2, # number of epochs to train
        "optimizer_cls": torch.optim.Adam, # optimization algorithm to use
        "optimizer_params": {'lr': 1e-3}, # initialization parameters for this algorithm
        "criterions": {"CE": torch.nn.CrossEntropyLoss()}, # the loss function
        "lr_sched_cls": None,  # the learning rate scheduling algorithm to use
        "lr_sched_params": {}, # the corresponding initialization parameters
        "metrics": {} # and some evaluation metrics
    }
})

# recreating the data managers with the batchsize of the params object
manager_train = BaseDataManager(dataset_train, params.nested_get("batch_size"), 1,
                                transforms=None, sampler_cls=RandomSampler,
                                n_process_loading=4)
manager_val = BaseDataManager(dataset_val, params.nested_get("batch_size"), 3,
                              transforms=None, sampler_cls=SequentialSampler,
                              n_process_loading=4)

Trainer¶

The delira.training.NetworkTrainer class provides functions to train a single network by passing attributes from your parameter object, a save_freq to specify how often your model should be saved (save_freq=1 indicates every epoch, save_freq=2 every second epoch etc.) and gpu_ids. If you don’t pass any ids at all, your network will be trained on CPU (and probably take a lot of time). If you specify 1 id, the network will be trained on the GPU with the corresponding index and if you pass multiple gpu_ids your network will be trained on multiple GPUs in parallel.

Note: The GPU indices are refering to the devices listed in CUDA_VISIBLE_DEVICES. E.g if CUDA_VISIBLE_DEVICES lists GPUs 3, 4, 5 then gpu_id 0 will be the index for GPU 3 etc.

Note: training on multiple GPUs is not recommended for easy and small networks, since for these networks the synchronization overhead is far greater than the parallelization benefit.

Training your network might look like this:

from delira.training import PyTorchNetworkTrainer
from delira.models.classification import ClassificationNetworkBasePyTorch

# path where checkpoints should be saved
save_path = "./results/checkpoints"

model = ClassificationNetworkBasePyTorch(in_channels=1, n_outputs=10)

trainer = PyTorchNetworkTrainer(network=model,
                                save_path=save_path,
                                criterions=params.nested_get("criterions"),
                                optimizer_cls=params.nested_get("optimizer_cls"),
                                optimizer_params=params.nested_get("optimizer_params"),
                                metrics=params.nested_get("metrics"),
                                lr_scheduler_cls=params.nested_get("lr_sched_cls"),
                                lr_scheduler_params=params.nested_get("lr_sched_params"),
                                gpu_ids=[0]
                        )

#trainer.train(params.nested_get("num_epochs"), manager_train, manager_val)

Experiment¶

The delira.training.AbstractExperiment class needs an experiment name, a path to save it’s results to, a parameter object, a model class and the keyword arguments to create an instance of this class. It provides methods to perform a single training and also a method for running a kfold-cross validation. In order to create it, you must choose the PyTorchExperiment, which is basically just a subclass of the AbstractExperiment to provide a general setup for PyTorch modules. Running an experiment could look like this:

from delira.training import PyTorchExperiment
from delira.training.train_utils import create_optims_default_pytorch

# Add model parameters to Parameter class
params.fixed.model = {"in_channels": 1, "n_outputs": 10}

experiment = PyTorchExperiment(params=params,
                               model_cls=ClassificationNetworkBasePyTorch,
                               name="TestExperiment",
                               save_path="./results",
                               optim_builder=create_optims_default_pytorch,
                               gpu_ids=[0])

experiment.run(manager_train, manager_val)

An Experiment is the most abstract (and recommended) way to define, train and validate your network.

Logging¶

Previous class and function definitions used pythons’s logging library. As extensions for this library delira provides a package (delira.logging) containing handlers to realize different logging methods.

To use these handlers simply add them to your logger like this:

logger.addHandler(logging.StreamHandler())

Nowadays, delira mainly relies on trixi for logging and provides only a MultiStreamHandler and a TrixiHandler, which is a binding to trixi’s loggers and integrates them into the python logging module

`MultiStreamHandler`¶

The MultiStreamHandler accepts an arbitrary number of streams during initialization and writes the message to all of it’s streams during logging.

Logging with `Visdom` - The `trixi` Loggers¶

`Visdom <https://github.com/facebookresearch/visdom>`__ is a tool designed to visualize your logs. To use this tool you need to open a port on the machine you want to train on via visdom -port YOUR_PORTNUMBER Afterwards just add the handler of your choice to the logger. For more detailed information and customization have a look at this website.

Logging the scalar tensors containing 1, 2, 3, 4 (at the beginning; will increase to show epochwise logging) with the corresponding keys "one", "two", "three", "four" and two random images with the keys "prediction" and "groundtruth" would look like this:

NUM_ITERS = 4

# import logging handler and logging module
from delira.logging import TrixiHandler
from trixi.logger import PytorchVisdomLogger
import logging

# configure logging module (and root logger)
logger_kwargs = {
    'name': 'test_env', # name of loggin environment
    'port': 9999 # visdom port to connect to
}
logger_cls = PytorchVisdomLogger

# configure logging module (and root logger)
logging.basicConfig(level=logging.INFO,
                    handlers=[TrixiHandler(logger_cls, **logger_kwargs)])
# derive logger from root logger
# (don't do `logger = logging.Logger("...")` since this will create a new
# logger which is unrelated to the root logger
logger = logging.getLogger("Test Logger")

# create dict containing the scalar numbers as torch.Tensor
scalars = {"one": torch.Tensor([1]),
           "two": torch.Tensor([2]),
           "three": torch.Tensor([3]),
           "four": torch.Tensor([4])}

# create dict containing the images as torch.Tensor
# pytorch awaits tensor dimensionality of
# batchsize x image channels x height x width
images = {"prediction": torch.rand(1, 3, 224, 224),
          "groundtruth": torch.rand(1, 3, 224, 224)}

# Simulate 4 Epochs
for i in range(4*NUM_ITERS):
    logger.info({"image_grid": {"images": images["prediction"], "name": "predictions"}})

    for key, val_tensor in scalars.items():
        logger.info({"value": {"value": val_tensor.item(), "name": key}})
        scalars[key] += 1

More Examples¶

More Examples can be found in * the classification example * the 2d segmentation example * the 3d segmentation example * the generative adversarial example

# Dataset Guide (incl. Integration to new API) With Delira v0.3.2 a new dataset API was introduced to allow for more flexibility and add some features. This notebook shows the difference between the new and the old API and provides some examples for newly added features.

## Overview Old API The old dataset API was based on the assumption that the underlying structure of the data can be described as followed: * root * sample1 * img1 * img2 * label * sample2 * img1 * img2 * label * …

A single sample was constructed from multiple images which are all located in the same subdirectory. The corresponding signature of the AbstractDataset was given by data_path, load_fn, img_extensions, gt_extensions. While most datasets need a load_fn to load a single sample and a data_path to the root directory, img_extensionsand gt_exntensions were often unsed. As a consequence a new dataset needed to be created which initialises the unused variables with arbitrary values.

## Overview New API The new dataset API was refactored to a more general approach where only a data_path to the root directory and a load_fn for a single sample need to be provided. A simple loading function (load_fn) to generate random data independent from the given path might be realized as below.

import numpy as np


def load_random_data(path: str) -> dict:
    """Load random data

    Parameters
    ----------
    path : str
        path to sample (not used in this example)

    Returns
    -------
    dict
        return data inside a dict
    """
    return {
        'data': np.random.rand(3, 512, 512),
        'label': np.random.randint(0, 10),
        'path': path,
    }

When used with the provided BaseDatasets, the return value of the load function is not limited to dictionaries and might be of any type which can be added to a list with the append method.

### New Datasets Some basic datasets are already implemented inside Delira and should be suitable for most cases. The BaseCacheDataset saves all samples inside the RAM and thus can only be used if everything fits inside the memory. ´BaseLazyDataset´ loads the individual samples on time when they are needed, but might lead to slower training due to the additional loading time.

from delira.data_loading import BaseCacheDataset, BaseLazyDataset


# because `load_random_data` does not use the path argument, they can have
# arbitrary values in this example
paths = list(range(10))

# create case dataset
cached_set = BaseCacheDataset(paths, load_random_data)

# create lazy dataset
lazy_set = BaseLazyDataset(paths, load_random_data)

# print cached data
print(cached_set[0].keys())

# print lazy data
print(lazy_set[0].keys())

In the above example a list of multiple paths is used as the data_path. load_fn is called for every element inside the provided list (can be any iterator). If data_path is a single string, it is assumed to be the path to the root directory. In this case, load_fnis called for every element inside the root directory.

Sometimes, a single file/folder contains multiple samples. BaseExtendCacheDataset uses the extend function to add elements to the internal list. Thus it is assumed that load_fn provides an iterable object, where eacht item represents a single data sample.

AbstractDataset is now iterable and can be used directly in combination with for loops.

for cs in cached_set:
    print(cs["path"])

## New Utility Function (Integration to new API) The behavior of the old API can be replicated with the LoadSample, LoadSampleLabelfunctions. LoadSample assumes that all needed images and the label (for a single sample) are located in a directory. Both functions return a dictionary containing the loaded data. sample_ext maps keys to iterables. Each iterable defines the names of the images which should be loaded from the directory. ´sample_fn´ is used to load the images which are than stacked inside a single array.

from delira.data_loading import LoadSample, LoadSampleLabel


def load_random_array(path: str):
    """Return random data

    Parameters
    ----------
    path : str
        path to image

    Returns
    -------
    np.ndarray
        loaded data
    """
    return np.random.rand(128, 128)


# define the function to load a single sample from a directory
load_fn = LoadSample(
    sample_ext={
        # load 3 data channels
        'data': ['red.png', 'green.png', 'blue.png'],
        # load a singel segmentation channel
        'seg': ['seg.png']
    },
    sample_fn=load_random_array,
    # optionally: assign individual keys a datatype
    dtype={"data": "float", "seg": "uint8"},
    # optioanlly: normalize individual samples
    normalize=["data"])

# Note: in general the function should be called with the path of the
# directory where the imags are located
sample0 = load_fn(".")

print("data shape: {}".format(sample0["data"].shape))
print("segmentation shape: {}".format(sample0["seg"].shape))
print("data type: {}".format(sample0["data"].dtype))
print("segmentation type: {}".format(sample0["seg"].dtype))
print("data min value: {}".format(sample0["data"].min()))
print("data max value: {}".format(sample0["data"].max()))

By default the range is normalized to (-1, 1), but norm_fn can be changed to achieve other normalization schemes. Some examples are included in delira.data_loading.load_utils.

LoadSampleLabel takes an additional argument for the label and a function to load a label. This functions can be used in combination with the provided BaseDatasets to replicate (and extend) the old API.

Classification with Delira - A very short introduction¶

Author: Justus Schock

Date: 04.12.2018

This Example shows how to set up a basic classification PyTorch experiment and Visdom Logging Environment.

Let’s first setup the essential hyperparameters. We will use delira’s Parameters-class for this:

logger = None
import torch
from delira.training import Parameters
params = Parameters(fixed_params={
    "model": {
        "in_channels": 1,
        "n_outputs": 10
    },
    "training": {
        "batch_size": 64, # batchsize to use
        "num_epochs": 10, # number of epochs to train
        "optimizer_cls": torch.optim.Adam, # optimization algorithm to use
        "optimizer_params": {'lr': 1e-3}, # initialization parameters for this algorithm
        "losses": {"CE": torch.nn.CrossEntropyLoss()}, # the loss function
        "lr_sched_cls": None,  # the learning rate scheduling algorithm to use
        "lr_sched_params": {}, # the corresponding initialization parameters
        "metrics": {} # and some evaluation metrics
    }
})

Since we did not specify any metric, only the CrossEntropyLoss will be calculated for each batch. Since we have a classification task, this should be sufficient. We will train our network with a batchsize of 64 by using Adam as optimizer of choice.

Logging and Visualization¶

To get a visualization of our results, we should monitor them somehow. For logging we will use Visdom. To start a visdom server you need to execute the following command inside an environment which has visdom installed:

visdom -port=9999

This will start a visdom server on port 9999 of your machine and now we can start to configure our logging environment. To view your results you can open http://localhost:9999 in your browser.

from trixi.logger import PytorchVisdomLogger
from delira.logging import TrixiHandler
import logging

logger_kwargs = {
    'name': 'ClassificationExampleLogger', # name of our logging environment
    'port': 9999 # port on which our visdom server is alive
}

logger_cls = PytorchVisdomLogger

# configure logging module (and root logger)
logging.basicConfig(level=logging.INFO,
                    handlers=[TrixiHandler(logger_cls, **logger_kwargs)])


# derive logger from root logger
# (don't do `logger = logging.Logger("...")` since this will create a new
# logger which is unrelated to the root logger
logger = logging.getLogger("Test Logger")

Since a single visdom server can run multiple environments, we need to specify a (unique) name for our environment and need to tell the logger, on which port it can find the visdom server.

Data Preparation¶

Loading¶

Next we will create a small train and validation set (based on torchvision MNIST):

from delira.data_loading import TorchvisionClassificationDataset

dataset_train = TorchvisionClassificationDataset("mnist", # which dataset to use
                                                 train=True, # use trainset
                                                 img_shape=(224, 224) # resample to 224 x 224 pixels
                                                )
dataset_val = TorchvisionClassificationDataset("mnist",
                                               train=False,
                                               img_shape=(224, 224)
                                              )

Augmentation¶

For Data-Augmentation we will apply a few transformations:

from batchgenerators.transforms import RandomCropTransform, \
                                        ContrastAugmentationTransform, Compose
from batchgenerators.transforms.spatial_transforms import ResizeTransform
from batchgenerators.transforms.sample_normalization_transforms import MeanStdNormalizationTransform

transforms = Compose([
    RandomCropTransform(200), # Perform Random Crops of Size 200 x 200 pixels
    ResizeTransform(224), # Resample these crops back to 224 x 224 pixels
    ContrastAugmentationTransform(), # randomly adjust contrast
    MeanStdNormalizationTransform(mean=[0.5], std=[0.5])])

With these transformations we can now wrap our datasets into datamanagers:

from delira.data_loading import BaseDataManager, SequentialSampler, RandomSampler

manager_train = BaseDataManager(dataset_train, params.nested_get("batch_size"),
                                transforms=transforms,
                                sampler_cls=RandomSampler,
                                n_process_augmentation=4)

manager_val = BaseDataManager(dataset_val, params.nested_get("batch_size"),
                              transforms=transforms,
                              sampler_cls=SequentialSampler,
                              n_process_augmentation=4)

Training¶

After we have done that, we can finally specify our experiment and run it. We will therfore use the already implemented ClassificationNetworkBasePyTorch which is basically a ResNet18:

import warnings
warnings.simplefilter("ignore", UserWarning) # ignore UserWarnings raised by dependency code
warnings.simplefilter("ignore", FutureWarning) # ignore FutureWarnings raised by dependency code


from delira.training import PyTorchExperiment
from delira.training.train_utils import create_optims_default_pytorch
from delira.models.classification import ClassificationNetworkBasePyTorch

if logger is not None:
    logger.info("Init Experiment")
experiment = PyTorchExperiment(params, ClassificationNetworkBasePyTorch,
                               name="ClassificationExample",
                               save_path="./tmp/delira_Experiments",
                               optim_builder=create_optims_default_pytorch,
                               gpu_ids=[0])
experiment.save()

model = experiment.run(manager_train, manager_val)

Congratulations, you have now trained your first Classification Model using delira, we will now predict a few samples from the testset to show, that the networks predictions are valid:

import numpy as np
from tqdm.auto import tqdm # utility for progress bars

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # set device (use GPU if available)
model = model.to(device) # push model to device
preds, labels = [], []

with torch.no_grad():
    for i in tqdm(range(len(dataset_val))):
        img = dataset_val[i]["data"] # get image from current batch
        img_tensor = torch.from_numpy(img).unsqueeze(0).to(device).to(torch.float) # create a tensor from image, push it to device and add batch dimension
        pred_tensor = model(img_tensor) # feed it through the network
        pred = pred_tensor.argmax(1).item() # get index with maximum class confidence
        label = np.asscalar(dataset_val[i]["label"]) # get label from batch
        if i % 1000 == 0:
            print("Prediction: %d \t label: %d" % (pred, label)) # print result
        preds.append(pred)
        labels.append(label)

# calculate accuracy
accuracy = (np.asarray(preds) == np.asarray(labels)).sum() / len(preds)
print("Accuracy: %.3f" % accuracy)

Generative Adversarial Nets with Delira - A very short introduction¶

Author: Justus Schock

Date: 04.12.2018

This Example shows how to set up a basic GAN PyTorch experiment and Visdom Logging Environment.

HyperParameters¶

Let’s first setup the essential hyperparameters. We will use delira’s Parameters-class for this:

logger = None
import torch
from delira.training import Parameters
params = Parameters(fixed_params={
    "model": {
        "n_channels": 1,
        "noise_length": 10
    },
    "training": {
        "batch_size": 64, # batchsize to use
        "num_epochs": 10, # number of epochs to train
        "optimizer_cls": torch.optim.Adam, # optimization algorithm to use
        "optimizer_params": {'lr': 1e-3}, # initialization parameters for this algorithm
        "losses": {"L1": torch.nn.L1Loss()}, # the loss function
        "lr_sched_cls": None,  # the learning rate scheduling algorithm to use
        "lr_sched_params": {}, # the corresponding initialization parameters
        "metrics": {} # and some evaluation metrics
    }
})

Since we specified torch.nn.L1Loss as criterion and torch.nn.MSELoss as metric, they will be both calculated for each batch, but only the criterion will be used for backpropagation. Since we have a simple generative task, this should be sufficient. We will train our network with a batchsize of 64 by using Adam as optimizer of choice.

Logging and Visualization¶

To get a visualization of our results, we should monitor them somehow. For logging we will use Visdom. To start a visdom server you need to execute the following command inside an environment which has visdom installed:

visdom -port=9999

This will start a visdom server on port 9999 of your machine and now we can start to configure our logging environment. To view your results you can open http://localhost:9999 in your browser.

from trixi.logger import PytorchVisdomLogger
from delira.logging import TrixiHandler
import logging

logger_kwargs = {
    'name': 'GANExampleLogger', # name of our logging environment
    'port': 9999 # port on which our visdom server is alive
}

logger_cls = PytorchVisdomLogger

# configure logging module (and root logger)
logging.basicConfig(level=logging.INFO,
                    handlers=[TrixiHandler(logger_cls, **logger_kwargs)])


# derive logger from root logger
# (don't do `logger = logging.Logger("...")` since this will create a new
# logger which is unrelated to the root logger
logger = logging.getLogger("Test Logger")

Since a single visdom server can run multiple environments, we need to specify a (unique) name for our environment and need to tell the logger, on which port it can find the visdom server.

Data Preparation¶

Loading¶

Next we will create a small train and validation set (based on torchvision MNIST):

from delira.data_loading import TorchvisionClassificationDataset

dataset_train = TorchvisionClassificationDataset("mnist", # which dataset to use
                                                 train=True, # use trainset
                                                 img_shape=(224, 224) # resample to 224 x 224 pixels
                                                )
dataset_val = TorchvisionClassificationDataset("mnist",
                                               train=False,
                                               img_shape=(224, 224)
                                              )

Augmentation¶

For Data-Augmentation we will apply a few transformations:

from batchgenerators.transforms import RandomCropTransform, \
                                        ContrastAugmentationTransform, Compose
from batchgenerators.transforms.spatial_transforms import ResizeTransform
from batchgenerators.transforms.sample_normalization_transforms import MeanStdNormalizationTransform

transforms = Compose([
    RandomCropTransform(200), # Perform Random Crops of Size 200 x 200 pixels
    ResizeTransform(224), # Resample these crops back to 224 x 224 pixels
    ContrastAugmentationTransform(), # randomly adjust contrast
    MeanStdNormalizationTransform(mean=[0.5], std=[0.5])])

With these transformations we can now wrap our datasets into datamanagers:

from delira.data_loading import BaseDataManager, SequentialSampler, RandomSampler

manager_train = BaseDataManager(dataset_train, params.nested_get("batch_size"),
                                transforms=transforms,
                                sampler_cls=RandomSampler,
                                n_process_augmentation=4)

manager_val = BaseDataManager(dataset_val, params.nested_get("batch_size"),
                              transforms=transforms,
                              sampler_cls=SequentialSampler,
                              n_process_augmentation=4)

Training¶

After we have done that, we can finally specify our experiment and run it. We will therfore use the already implemented GenerativeAdversarialNetworkBasePyTorch which is basically a vanilla DCGAN:

import warnings
warnings.simplefilter("ignore", UserWarning) # ignore UserWarnings raised by dependency code
warnings.simplefilter("ignore", FutureWarning) # ignore FutureWarnings raised by dependency code


from delira.training import PyTorchExperiment
from delira.training.train_utils import create_optims_gan_default_pytorch
from delira.models.gan import GenerativeAdversarialNetworkBasePyTorch

if logger is not None:
    logger.info("Init Experiment")
experiment = PyTorchExperiment(params, GenerativeAdversarialNetworkBasePyTorch,
                               name="GANExample",
                               save_path="./tmp/delira_Experiments",
                               optim_builder=create_optims_gan_default_pytorch,
                               gpu_ids=[0])
experiment.save()

model = experiment.run(manager_train, manager_val)

Congratulations, you have now trained your first Generative Adversarial Model using delira.

Segmentation in 2D using U-Nets with Delira - A very short introduction¶

Author: Justus Schock, Alexander Moriz

Date: 17.12.2018

This Example shows how use the U-Net implementation in Delira with PyTorch.

Let’s first setup the essential hyperparameters. We will use delira’s Parameters-class for this:

logger = None
import torch
from delira.training import Parameters
params = Parameters(fixed_params={
    "model": {
        "in_channels": 1,
        "num_classes": 4
    },
    "training": {
        "batch_size": 64, # batchsize to use
        "num_epochs": 10, # number of epochs to train
        "optimizer_cls": torch.optim.Adam, # optimization algorithm to use
        "optimizer_params": {'lr': 1e-3}, # initialization parameters for this algorithm
        "losses": {"CE": torch.nn.CrossEntropyLoss()}, # the loss function
        "lr_sched_cls": None,  # the learning rate scheduling algorithm to use
        "lr_sched_params": {}, # the corresponding initialization parameters
        "metrics": {} # and some evaluation metrics
    }
})

Since we did not specify any metric, only the CrossEntropyLoss will be calculated for each batch. Since we have a classification task, this should be sufficient. We will train our network with a batchsize of 64 by using Adam as optimizer of choice.

Logging and Visualization¶

To get a visualization of our results, we should monitor them somehow. For logging we will use Visdom. To start a visdom server you need to execute the following command inside an environment which has visdom installed:

visdom -port=9999

This will start a visdom server on port 9999 of your machine and now we can start to configure our logging environment. To view your results you can open http://localhost:9999 in your browser.

from trixi.logger import PytorchVisdomLogger
from delira.logging import TrixiHandler
import logging

logger_kwargs = {
    'name': 'ClassificationExampleLogger', # name of our logging environment
    'port': 9999 # port on which our visdom server is alive
}

logger_cls = PytorchVisdomLogger

# configure logging module (and root logger)
logging.basicConfig(level=logging.INFO,
                    handlers=[TrixiHandler(logger_cls, **logger_kwargs)])


# derive logger from root logger
# (don't do `logger = logging.Logger("...")` since this will create a new
# logger which is unrelated to the root logger
logger = logging.getLogger("Test Logger")

Since a single visdom server can run multiple environments, we need to specify a (unique) name for our environment and need to tell the logger, on which port it can find the visdom server.

Data Praparation¶

Loading¶

Next we will create a small train and validation set (in this case they will be the same to show the overfitting capability of the UNet).

Our data is a brain MR-image thankfully provided by the FSL in their introduction.

We first download the data and extract the T1 image and the corresponding segmentation:

from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen

resp = urlopen("http://www.fmrib.ox.ac.uk/primers/intro_primer/ExBox3/ExBox3.zip")
zipfile = ZipFile(BytesIO(resp.read()))
#zipfile_list = zipfile.namelist()
#print(zipfile_list)
img_file = zipfile.extract("ExBox3/T1_brain.nii.gz")
mask_file = zipfile.extract("ExBox3/T1_brain_seg.nii.gz")

Now, we load the image and the mask (they are both 3D), convert them to a 32-bit floating point numpy array and ensure, they have the same shape (i.e. that for each voxel in the image, there is a voxel in the mask):

import SimpleITK as sitk
import numpy as np

# load image and mask
img = sitk.GetArrayFromImage(sitk.ReadImage(img_file))
img = img.astype(np.float32)
mask = mask = sitk.GetArrayFromImage(sitk.ReadImage(mask_file))
mask = mask.astype(np.float32)

assert mask.shape == img.shape
print(img.shape)

By querying the unique values in the mask, we get the following:

np.unique(mask)

This means, there are 4 classes (background and 3 types of tissue) in our sample.

Since we want to do a 2D segmentation, we extract a single slice out of the image and the mask (we choose slice 100 here) and plot it:

import matplotlib.pyplot as plt

# load single slice
img_slice = img[:, :, 100]
mask_slice = mask[:, :, 100]

# plot slices
plt.figure(1, figsize=(15,10))
plt.subplot(121)
plt.imshow(img_slice, cmap="gray")
plt.colorbar(fraction=0.046, pad=0.04)
plt.subplot(122)
plt.imshow(mask_slice, cmap="gray")
plt.colorbar(fraction=0.046, pad=0.04)
plt.show()

To load the data, we have to use a Dataset. The following defines a very simple dataset, accepting an image slice, a mask slice and the number of samples. It always returns the same sample until num_samples samples have been returned.

from delira.data_loading import AbstractDataset

class CustomDataset(AbstractDataset):
    def __init__(self, img, mask, num_samples=1000):
        super().__init__(None, None, None, None)
        self.data = {"data": img.reshape(1, *img.shape), "label": mask.reshape(1, *mask.shape)}
        self.num_samples = num_samples

    def __getitem__(self, index):
        return self.data

    def __len__(self):
        return self.num_samples

Now, we can finally instantiate our datasets:

dataset_train = CustomDataset(img_slice, mask_slice, num_samples=10000)
dataset_val = CustomDataset(img_slice, mask_slice, num_samples=1)

Augmentation¶

For Data-Augmentation we will apply a few transformations:

from batchgenerators.transforms import RandomCropTransform, \
                                        ContrastAugmentationTransform, Compose
from batchgenerators.transforms.spatial_transforms import ResizeTransform
from batchgenerators.transforms.sample_normalization_transforms import MeanStdNormalizationTransform

transforms = Compose([
    RandomCropTransform(150, label_key="label"), # Perform Random Crops of Size 150 x 150 pixels
    ResizeTransform(224, label_key="label"), # Resample these crops back to 224 x 224 pixels
    ContrastAugmentationTransform(), # randomly adjust contrast
    MeanStdNormalizationTransform(mean=[img_slice.mean()], std=[img_slice.std()])]) # use concrete values since we only have one sample (have to estimate it over whole dataset otherwise)

With these transformations we can now wrap our datasets into datamanagers:

from delira.data_loading import BaseDataManager, SequentialSampler, RandomSampler

manager_train = BaseDataManager(dataset_train, params.nested_get("batch_size"),
                                transforms=transforms,
                                sampler_cls=RandomSampler,
                                n_process_augmentation=4)

manager_val = BaseDataManager(dataset_val, params.nested_get("batch_size"),
                              transforms=transforms,
                              sampler_cls=SequentialSampler,
                              n_process_augmentation=4)

Training¶

After we have done that, we can finally specify our experiment and run it. We will therfore use the already implemented UNet2dPytorch:

import warnings
warnings.simplefilter("ignore", UserWarning) # ignore UserWarnings raised by dependency code
warnings.simplefilter("ignore", FutureWarning) # ignore FutureWarnings raised by dependency code


from delira.training import PyTorchExperiment
from delira.training.train_utils import create_optims_default_pytorch
from delira.models.segmentation import UNet2dPyTorch

if logger is not None:
    logger.info("Init Experiment")
experiment = PyTorchExperiment(params, UNet2dPyTorch,
                               name="Segmentation2dExample",
                               save_path="./tmp/delira_Experiments",
                               optim_builder=create_optims_default_pytorch,
                               gpu_ids=[0], mixed_precision=True)
experiment.save()

model = experiment.run(manager_train, manager_val)

Segmentation in 3D using U-Nets with Delira - A very short introduction¶

Author: Justus Schock, Alexander Moriz

Date: 17.12.2018

This Example shows how use the U-Net implementation in Delira with PyTorch.

Let’s first setup the essential hyperparameters. We will use delira’s Parameters-class for this:

logger = None
import torch
from delira.training import Parameters
params = Parameters(fixed_params={
    "model": {
        "in_channels": 1,
        "num_classes": 4
    },
    "training": {
        "batch_size": 64, # batchsize to use
        "num_epochs": 10, # number of epochs to train
        "optimizer_cls": torch.optim.Adam, # optimization algorithm to use
        "optimizer_params": {'lr': 1e-3}, # initialization parameters for this algorithm
        "losses": {"CE": torch.nn.CrossEntropyLoss()}, # the loss function
        "lr_sched_cls": None,  # the learning rate scheduling algorithm to use
        "lr_sched_params": {}, # the corresponding initialization parameters
        "metrics": {} # and some evaluation metrics
    }
})

Since we did not specify any metric, only the CrossEntropyLoss will be calculated for each batch. Since we have a classification task, this should be sufficient. We will train our network with a batchsize of 64 by using Adam as optimizer of choice.

Logging and Visualization¶

To get a visualization of our results, we should monitor them somehow. For logging we will use Visdom. To start a visdom server you need to execute the following command inside an environment which has visdom installed:

visdom -port=9999

This will start a visdom server on port 9999 of your machine and now we can start to configure our logging environment. To view your results you can open http://localhost:9999 in your browser.

from trixi.logger import PytorchVisdomLogger
from delira.logging import TrixiHandler
import logging

logger_kwargs = {
    'name': 'ClassificationExampleLogger', # name of our logging environment
    'port': 9999 # port on which our visdom server is alive
}

logger_cls = PytorchVisdomLogger

# configure logging module (and root logger)
logging.basicConfig(level=logging.INFO,
                    handlers=[TrixiHandler(logger_cls, **logger_kwargs)])


# derive logger from root logger
# (don't do `logger = logging.Logger("...")` since this will create a new
# logger which is unrelated to the root logger
logger = logging.getLogger("Test Logger")

Since a single visdom server can run multiple environments, we need to specify a (unique) name for our environment and need to tell the logger, on which port it can find the visdom server.

Data Praparation¶

Loading¶

Next we will create a small train and validation set (in this case they will be the same to show the overfitting capability of the UNet).

Our data is a brain MR-image thankfully provided by the FSL in their introduction.

We first download the data and extract the T1 image and the corresponding segmentation:

from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen

resp = urlopen("http://www.fmrib.ox.ac.uk/primers/intro_primer/ExBox3/ExBox3.zip")
zipfile = ZipFile(BytesIO(resp.read()))
#zipfile_list = zipfile.namelist()
#print(zipfile_list)
img_file = zipfile.extract("ExBox3/T1_brain.nii.gz")
mask_file = zipfile.extract("ExBox3/T1_brain_seg.nii.gz")

Now, we load the image and the mask (they are both 3D), convert them to a 32-bit floating point numpy array and ensure, they have the same shape (i.e. that for each voxel in the image, there is a voxel in the mask):

import SimpleITK as sitk
import numpy as np

# load image and mask
img = sitk.GetArrayFromImage(sitk.ReadImage(img_file))
img = img.astype(np.float32)
mask = mask = sitk.GetArrayFromImage(sitk.ReadImage(mask_file))
mask = mask.astype(np.float32)

assert mask.shape == img.shape
print(img.shape)

By querying the unique values in the mask, we get the following:

np.unique(mask)

This means, there are 4 classes (background and 3 types of tissue) in our sample.

To load the data, we have to use a Dataset. The following defines a very simple dataset, accepting an image slice, a mask slice and the number of samples. It always returns the same sample until num_samples samples have been returned.

from delira.data_loading import AbstractDataset

class CustomDataset(AbstractDataset):
    def __init__(self, img, mask, num_samples=1000):
        super().__init__(None, None, None, None)
        self.data = {"data": img.reshape(1, *img.shape), "label": mask.reshape(1, *mask.shape)}
        self.num_samples = num_samples

    def __getitem__(self, index):
        return self.data

    def __len__(self):
        return self.num_samples

Now, we can finally instantiate our datasets:

dataset_train = CustomDataset(img, mask, num_samples=10000)
dataset_val = CustomDataset(img, mask, num_samples=1)

Augmentation¶

For Data-Augmentation we will apply a few transformations:

from batchgenerators.transforms import ContrastAugmentationTransform, Compose
from batchgenerators.transforms.sample_normalization_transforms import MeanStdNormalizationTransform

transforms = Compose([
    ContrastAugmentationTransform(), # randomly adjust contrast
    MeanStdNormalizationTransform(mean=[img.mean()], std=[img.std()])]) # use concrete values since we only have one sample (have to estimate it over whole dataset otherwise)

With these transformations we can now wrap our datasets into datamanagers:

from delira.data_loading import BaseDataManager, SequentialSampler, RandomSampler

manager_train = BaseDataManager(dataset_train, params.nested_get("batch_size"),
                                transforms=transforms,
                                sampler_cls=RandomSampler,
                                n_process_augmentation=4)

manager_val = BaseDataManager(dataset_val, params.nested_get("batch_size"),
                              transforms=transforms,
                              sampler_cls=SequentialSampler,
                              n_process_augmentation=4)

Training¶

After we have done that, we can finally specify our experiment and run it. We will therfore use the already implemented UNet3dPytorch:

import warnings
warnings.simplefilter("ignore", UserWarning) # ignore UserWarnings raised by dependency code
warnings.simplefilter("ignore", FutureWarning) # ignore FutureWarnings raised by dependency code


from delira.training import PyTorchExperiment
from delira.training.train_utils import create_optims_default_pytorch
from delira.models.segmentation import UNet3dPyTorch

if logger:
    logger.info("Init Experiment")
experiment = PyTorchExperiment(params, UNet3dPyTorch,
                               name="Segmentation3dExample",
                               save_path="./tmp/delira_Experiments",
                               optim_builder=create_optims_default_pytorch,
                               gpu_ids=[0], mixed_precision=True)
experiment.save()

model = experiment.run(manager_train, manager_val)

API Documentation¶

Delira¶

Data Loading¶

This module provides Utilities to load the Data

Arbitrary Data¶

The following classes are implemented to work with every kind of data. You can use every framework you want to load your data, but the returned samples should be a dict of numpy ndarrays

Datasets¶

The Dataset the most basic class and implements the loading of your dataset elements. You can either load your data in a lazy way e.g. loading them just at the moment they are needed or you could preload them and cache them.

Datasets can be indexed by integers and return single samples.

To implement custom datasets you should derive the AbstractDataset

AbstractDataset¶

class AbstractDataset(data_path: str, load_fn: Callable)[source]¶

Bases: object

Base Class for Dataset

abstract _make_dataset(path: str)[source]¶

Create dataset

Parameters: path (str) – path to data samples
Returns: data: List of sample paths if lazy; List of samples if not
Return type: list

get_sample_from_index(index)[source]¶

Returns the data sample for a given index (without any loading if it would be necessary) This implements the base case and can be subclassed for index mappings. The actual loading behaviour (lazy or cached) should be implemented in __getitem__

See also

:method:ConcatDataset.get_sample_from_index :method:BaseLazyDataset.__getitem__ :method:BaseCacheDataset.__getitem__

Parameters: index (int) – index corresponding to targeted sample
Returns: sample corresponding to given index
Return type: Any

get_subset(indices)[source]¶

Returns a Subset of the current dataset based on given indices

Parameters: indices (iterable) – valid indices to extract subset from current dataset
Returns: the subset
Return type: BlankDataset

train_test_split(*args, **kwargs)[source]¶

split dataset into train and test data

Parameters

*args – positional arguments of train_test_split
**kwargs – keyword arguments of train_test_split

Returns

BlankDataset – train dataset
BlankDataset – test dataset

See also

sklearn.model_selection.train_test_split

BaseLazyDataset¶

class BaseLazyDataset(data_path: Union[str, list], load_fn: Callable, **load_kwargs)[source]¶

Bases: delira.data_loading.dataset.AbstractDataset

Dataset to load data in a lazy way

_make_dataset(path: Union[str, list])[source]¶

Helper Function to make a dataset containing paths to all images in a certain directory

Parameters: path (str or list) – path to data samples
Returns: list of sample paths
Return type: list
Raises: AssertionError – if path is not a valid directory

get_sample_from_index(index)¶

Returns the data sample for a given index (without any loading if it would be necessary) This implements the base case and can be subclassed for index mappings. The actual loading behaviour (lazy or cached) should be implemented in __getitem__

See also

:method:ConcatDataset.get_sample_from_index :method:BaseLazyDataset.__getitem__ :method:BaseCacheDataset.__getitem__

Parameters: index (int) – index corresponding to targeted sample
Returns: sample corresponding to given index
Return type: Any

get_subset(indices)¶

Returns a Subset of the current dataset based on given indices

Parameters: indices (iterable) – valid indices to extract subset from current dataset
Returns: the subset
Return type: BlankDataset

train_test_split(*args, **kwargs)¶

split dataset into train and test data

Parameters

*args – positional arguments of train_test_split
**kwargs – keyword arguments of train_test_split

Returns

BlankDataset – train dataset
BlankDataset – test dataset

See also

sklearn.model_selection.train_test_split

BaseCacheDataset¶

class BaseCacheDataset(data_path: Union[str, list], load_fn: Callable, **load_kwargs)[source]¶

Bases: delira.data_loading.dataset.AbstractDataset

Dataset to preload and cache data

Notes

data needs to fit completely into RAM!

_make_dataset(path: Union[str, list])[source]¶

Helper Function to make a dataset containing all samples in a certain directory

Parameters: path (str or list) – if data_path is a string, _sample_fn is called for all items inside the specified directory if data_path is a list, _sample_fn is called for elements in the list
Returns: list of items which where returned from _sample_fn (typically dict)
Return type: list
Raises: AssertionError – if path is not a list and is not a valid directory

get_sample_from_index(index)¶

Returns the data sample for a given index (without any loading if it would be necessary) This implements the base case and can be subclassed for index mappings. The actual loading behaviour (lazy or cached) should be implemented in __getitem__

See also

:method:ConcatDataset.get_sample_from_index :method:BaseLazyDataset.__getitem__ :method:BaseCacheDataset.__getitem__

Parameters: index (int) – index corresponding to targeted sample
Returns: sample corresponding to given index
Return type: Any

get_subset(indices)¶

Returns a Subset of the current dataset based on given indices

Parameters: indices (iterable) – valid indices to extract subset from current dataset
Returns: the subset
Return type: BlankDataset

train_test_split(*args, **kwargs)¶

split dataset into train and test data

Parameters

*args – positional arguments of train_test_split
**kwargs – keyword arguments of train_test_split

Returns

BlankDataset – train dataset
BlankDataset – test dataset

See also

sklearn.model_selection.train_test_split

BaseExtendCacheDataset¶

class BaseExtendCacheDataset(data_path: Union[str, list], load_fn: Callable, **load_kwargs)[source]¶

Bases: delira.data_loading.dataset.BaseCacheDataset

Dataset to preload and cache data. Function to load sample is expected to return an iterable which can contain multiple samples

Notes

data needs to fit completely into RAM!

_make_dataset(path: Union[str, list])[source]¶

Helper Function to make a dataset containing all samples in a certain directory

Parameters: path (str or iterable) – if data_path is a string, _sample_fn is called for all items inside the specified directory if data_path is a list, _sample_fn is called for elements in the list
Returns: list of items which where returned from _sample_fn (typically dict)
Return type: list
Raises: AssertionError – if path is not a list and is not a valid directory

get_sample_from_index(index)¶

Returns the data sample for a given index (without any loading if it would be necessary) This implements the base case and can be subclassed for index mappings. The actual loading behaviour (lazy or cached) should be implemented in __getitem__

See also

:method:ConcatDataset.get_sample_from_index :method:BaseLazyDataset.__getitem__ :method:BaseCacheDataset.__getitem__

Parameters: index (int) – index corresponding to targeted sample
Returns: sample corresponding to given index
Return type: Any

get_subset(indices)¶

Returns a Subset of the current dataset based on given indices

Parameters: indices (iterable) – valid indices to extract subset from current dataset
Returns: the subset
Return type: BlankDataset

train_test_split(*args, **kwargs)¶

split dataset into train and test data

Parameters

*args – positional arguments of train_test_split
**kwargs – keyword arguments of train_test_split

Returns

BlankDataset – train dataset
BlankDataset – test dataset

See also

sklearn.model_selection.train_test_split

ConcatDataset¶

class ConcatDataset(*datasets)[source]¶

Bases: delira.data_loading.dataset.AbstractDataset

abstract _make_dataset(path: str)¶

Create dataset

Parameters: path (str) – path to data samples
Returns: data: List of sample paths if lazy; List of samples if not
Return type: list

get_sample_from_index(index)[source]¶

Returns the data sample for a given index (without any loading if it would be necessary) This method implements the index mapping of a global index to the subindices for each dataset. The actual loading behaviour (lazy or cached) should be implemented in __getitem__

See also

:method:AbstractDataset.get_sample_from_index :method:BaseLazyDataset.__getitem__ :method:BaseCacheDataset.__getitem__

Parameters: index (int) – index corresponding to targeted sample
Returns: sample corresponding to given index
Return type: Any

get_subset(indices)¶

Returns a Subset of the current dataset based on given indices

Parameters: indices (iterable) – valid indices to extract subset from current dataset
Returns: the subset
Return type: BlankDataset

train_test_split(*args, **kwargs)¶

split dataset into train and test data

Parameters

*args – positional arguments of train_test_split
**kwargs – keyword arguments of train_test_split

Returns

BlankDataset – train dataset
BlankDataset – test dataset

See also

sklearn.model_selection.train_test_split

BlankDataset¶

Nii3DLazyDataset¶

Nii3DCacheDataset¶

TorchvisionClassificationDataset:¶

class TorchvisionClassificationDataset(dataset, root='/tmp/', train=True, download=True, img_shape=(28, 28), one_hot=False, **kwargs)[source]¶

Bases: delira.data_loading.dataset.AbstractDataset

Wrapper for torchvision classification datasets to provide consistent API

_make_dataset(dataset, **kwargs)[source]¶

Create the actual dataset

Parameters

dataset (str) – Defines the dataset to use. must be one of [‘mnist’, ‘emnist’, ‘fashion_mnist’, ‘cifar10’, ‘cifar100’]
**kwargs – Additional keyword arguments passed to the torchvision dataset class for initialization

Returns

actual Dataset

Return type

torchvision.Dataset

Raises

KeyError – Dataset string does not specify a valid dataset

get_sample_from_index(index)¶

Returns the data sample for a given index (without any loading if it would be necessary) This implements the base case and can be subclassed for index mappings. The actual loading behaviour (lazy or cached) should be implemented in __getitem__

See also

:method:ConcatDataset.get_sample_from_index :method:BaseLazyDataset.__getitem__ :method:BaseCacheDataset.__getitem__

Parameters: index (int) – index corresponding to targeted sample
Returns: sample corresponding to given index
Return type: Any

get_subset(indices)¶

Returns a Subset of the current dataset based on given indices

Parameters: indices (iterable) – valid indices to extract subset from current dataset
Returns: the subset
Return type: BlankDataset

train_test_split(*args, **kwargs)¶

split dataset into train and test data

Parameters

*args – positional arguments of train_test_split
**kwargs – keyword arguments of train_test_split

Returns

BlankDataset – train dataset
BlankDataset – test dataset

See also

sklearn.model_selection.train_test_split

Dataloader¶

The Dataloader wraps the dataset and combines them with a sampler (see below) to combine single samples to whole batches.

ToDo: add flow chart diagramm

BaseDataLoader¶

class BaseDataLoader(dataset: delira.data_loading.dataset.AbstractDataset, batch_size=1, num_batches=None, seed=1, sampler=None)[source]¶

Bases: batchgenerators.dataloading.data_loader.SlimDataLoaderBase

Class to create a data batch out of data samples

_get_sample(index)[source]¶

Helper functions which returns an element of the dataset

Parameters: index (int) – index specifying which sample to return
Returns: Returned Data
Return type: dict

generate_train_batch()[source]¶

Generate Indices which behavior based on self.sampling gets data based on indices

Returns: data and labels
Return type: dict
Raises: StopIteration – If the maximum number of batches has been generated

Datamanager¶

The datamanager wraps a dataloader and combines it with augmentations and multiprocessing.

BaseDataManager¶

class BaseDataManager(data, batch_size, n_process_augmentation, transforms, sampler_cls=<class 'delira.data_loading.sampler.sequential_sampler.SequentialSampler'>, sampler_kwargs={}, data_loader_cls=None, dataset_cls=None, load_fn=<function default_load_fn_2d>, from_disc=True, **kwargs)[source]¶

Bases: object

Class to Handle Data Creates Dataset , Dataloader and BatchGenerator

property batch_size¶

Property to access the batchsize

Returns: the batchsize
Return type: int

property data_loader_cls¶

Property to access the current data loader class

Returns: Subclass of SlimDataLoaderBase
Return type: type

property dataset¶

Property to access the current dataset

Returns: the current dataset
Return type: AbstractDataset

get_batchgen(seed=1)[source]¶

Create DataLoader and Batchgenerator

Parameters: seed (int) – seed for Random Number Generator
Returns: Batchgenerator
Return type: Augmenter
Raises: AssertionError – BaseDataManager.n_batches is smaller than or equal to zero

get_subset(indices)[source]¶

Returns a Subset of the current datamanager based on given indices

Parameters: indices (iterable) – valid indices to extract subset from current dataset
Returns: manager containing the subset
Return type: BaseDataManager

property n_batches¶

Returns Number of Batches based on batchsize and number of samples

Returns: Number of Batches
Return type: int
Raises: AssertionError – BaseDataManager.n_samples is smaller than or equal to zero

property n_process_augmentation¶

Property to access the number of augmentation processes

Returns: number of augmentation processes
Return type: int

property n_samples¶

Number of Samples

Returns: Number of Samples
Return type: int

property sampler¶

Property to access the current sampler

Returns: the current sampler
Return type: AbstractSampler

train_test_split(*args, **kwargs)[source]¶

Calls :method:`AbstractDataset.train_test_split` and returns a manager for each subset with same configuration as current manager

Parameters

*args – positional arguments for sklearn.model_selection.train_test_split
**kwargs – keyword arguments for sklearn.model_selection.train_test_split

property transforms¶

Property to access the current data transforms

Returns: The transformation, can either be None or an instance of AbstractTransform
Return type: None, AbstractTransform

update_state_from_dict(new_state: dict)[source]¶

Updates internal state and therfore the behavior from dict. If a key is not specified, the old attribute value will be used

Parameters

new_state (dict) –

The dict to update the state from. Valid keys are:

batch_size

n_process_augmentation

data_loader_cls

sampler

sampling_kwargs

transforms

If a key is not specified, the old value of the corresponding attribute will be used

Raises

KeyError – Invalid keys are specified

Utils¶

norm_range¶

norm_range(mode)[source]¶

Closure function for range normalization

Parameters: mode (str) – ‘-1,1’ normalizes data to range [-1, 1], while ‘0,1’ normalizes data to range [0, 1]
Returns: normalization function
Return type: callable

norm_zero_mean_unit_std¶

norm_zero_mean_unit_std(data)[source]¶

Return normalized data with mean 0, standard deviation 1

Parameters: data (np.nadarray) –
Returns: normalized data
Return type: np.ndarray

is_valid_image_file¶

is_valid_image_file(fname, img_extensions, gt_extensions)[source]¶

Helper Function to check wheter file is image file and has at least one label file

Parameters

fname (str) – filename of image path Returns
------- –
bool – is valid data sample

default_load_fn_2d¶

default_load_fn_2d(img_file, *label_files, img_shape, n_channels=1)[source]¶

loading single 2d sample with arbitrary number of samples

Parameters

img_file (string) – path to image file
label_files (list of strings) – paths to label files
img_shape (iterable) – shape of image
n_channels (int) – number of image channels

Returns

numpy.ndarray – image
Any – labels

LoadSample¶

class LoadSample(sample_ext: dict, sample_fn: collections.abc.Callable, dtype={}, normalize=(), norm_fn=<function norm_range.<locals>.norm_fn>, **kwargs)[source]¶

Bases: object

Provides a callable to load a single sample from multiple files in a folder

Nii-Data¶

Since delira aims to provide dataloading tools for medical data (which is often stored in Nii-Files), the following classes and functions provide a basic way to load data from nii-files:

load_nii¶

load_nii(path)[source]¶

Loads a single nii file :param path: path to nii file which should be loaded :type path: str

Returns: numpy array containing the loaded data
Return type: np.ndarray

BaseLabelGenerator¶

class BaseLabelGenerator(fpath)[source]¶

Bases: object

Base Class to load labels from json files

_load()[source]¶

Private Helper function to load the file

Returns: loaded values from file
Return type: Any

abstract get_labels()[source]¶

Abstractmethod to get labels from class

Raises: NotImplementedError – if not overwritten in subclass

load_sample_nii¶

load_sample_nii(files, label_load_cls)[source]¶

Load sample from multiple ITK files

Parameters

files (dict with keys img and label) – filenames of nifti files and label file
label_load_cls (class) – function to be used for label parsing

Returns

sample: dict with keys data and label containing images and label

Return type

dict

Raises

AssertionError – if img.max() is greater than 511 or smaller than 1

Sampler¶

Sampler define the way of iterating over the dataset and returning samples.

AbstractSampler¶

class AbstractSampler(indices=None)[source]¶

Bases: object

Class to define an abstract Sampling API

_check_batchsize(n_indices)[source]¶

Checks if the batchsize is valid (and truncates batches if necessary). Will also raise StopIteration if enough batches sampled

Parameters: n_indices (int) – number of indices to sample
Returns: number of indices to sample (truncated if necessary)
Return type: int
Raises: StopIteration – if enough batches sampled

abstract _get_indices(n_indices)[source]¶

Function to return a specific number of indices. Implements the actual sampling strategy.

Parameters: n_indices (int) – Number of indices to return
Returns: List with sampled indices
Return type: list

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset, **kwargs)[source]¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialzed sampler
Return type: AbstractSampler

LambdaSampler¶

class LambdaSampler(indices, sampling_fn)[source]¶

Bases: delira.data_loading.sampler.abstract_sampler.AbstractSampler

Implements Arbitrary Sampling methods specified by a function which takes the index_list and the number of indices to return

_check_batchsize(n_indices)¶

Checks if the batchsize is valid (and truncates batches if necessary). Will also raise StopIteration if enough batches sampled

Parameters: n_indices (int) – number of indices to sample
Returns: number of indices to sample (truncated if necessary)
Return type: int
Raises: StopIteration – if enough batches sampled

_get_indices(n_indices)[source]¶

Actual Sampling

Parameters: n_indices (int) – number of indices to return
Returns: list of sampled indices
Return type: list

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset, **kwargs)¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialzed sampler
Return type: AbstractSampler

RandomSampler¶

class RandomSampler(indices)[source]¶

Bases: delira.data_loading.sampler.abstract_sampler.AbstractSampler

Implements Random Sampling from whole Dataset

_check_batchsize(n_indices)¶

Checks if the batchsize is valid (and truncates batches if necessary). Will also raise StopIteration if enough batches sampled

Parameters: n_indices (int) – number of indices to sample
Returns: number of indices to sample (truncated if necessary)
Return type: int
Raises: StopIteration – if enough batches sampled

_get_indices(n_indices)[source]¶

Actual Sampling

Parameters: n_indices (int) – number of indices to return
Returns: list of sampled indices
Return type: list
Raises: StopIteration – If maximal number of samples is reached

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset, **kwargs)¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialzed sampler
Return type: AbstractSampler

PrevalenceRandomSampler¶

class PrevalenceRandomSampler(indices, shuffle_batch=True)[source]¶

Bases: delira.data_loading.sampler.abstract_sampler.AbstractSampler

Implements random Per-Class Sampling and ensures same number of samplers per batch for each class

_check_batchsize(n_indices)¶

Checks if the batchsize is valid (and truncates batches if necessary). Will also raise StopIteration if enough batches sampled

Parameters: n_indices (int) – number of indices to sample
Returns: number of indices to sample (truncated if necessary)
Return type: int
Raises: StopIteration – if enough batches sampled

_get_indices(n_indices)[source]¶

Actual Sampling

Parameters: n_indices (int) – number of indices to return
Returns: list of sampled indices
Return type: list
Raises: StopIteration – If maximal number of samples is reached

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset, **kwargs)[source]¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialized sampler
Return type: AbstractSampler

StoppingPrevalenceRandomSampler¶

class StoppingPrevalenceRandomSampler(indices, shuffle_batch=True)[source]¶

Bases: delira.data_loading.sampler.abstract_sampler.AbstractSampler

Implements random Per-Class Sampling and ensures same number of samplers per batch for each class; Stops if out of samples for smallest class

_check_batchsize(n_indices)[source]¶

Checks if batchsize is valid for all classes

Parameters: n_indices (int) – the number of samples to return
Returns: number of samples per class to return
Return type: dict

_get_indices(n_indices)[source]¶

Actual Sampling

Parameters: n_indices (int) – number of indices to return
Raises: StopIteration – If end of class indices is reached for one class:
Returns: list
Return type: list of sampled indices

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset, **kwargs)[source]¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialized sampler
Return type: AbstractSampler

SequentialSampler¶

class SequentialSampler(indices)[source]¶

Bases: delira.data_loading.sampler.abstract_sampler.AbstractSampler

Implements Sequential Sampling from whole Dataset

_check_batchsize(n_indices)¶

Checks if the batchsize is valid (and truncates batches if necessary). Will also raise StopIteration if enough batches sampled

Parameters: n_indices (int) – number of indices to sample
Returns: number of indices to sample (truncated if necessary)
Return type: int
Raises: StopIteration – if enough batches sampled

_get_indices(n_indices)[source]¶

Actual Sampling

Parameters: n_indices (int) – number of indices to return

:raises StopIteration : If end of dataset reached:

Returns: list of sampled indices
Return type: list

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset, **kwargs)¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialzed sampler
Return type: AbstractSampler

PrevalenceSequentialSampler¶

class PrevalenceSequentialSampler(indices, shuffle_batch=True)[source]¶

Bases: delira.data_loading.sampler.abstract_sampler.AbstractSampler

Implements Per-Class Sequential sampling and ensures same number of samples per batch for each class; If out of samples for one class: restart at first sample

_check_batchsize(n_indices)¶

Checks if the batchsize is valid (and truncates batches if necessary). Will also raise StopIteration if enough batches sampled

Parameters: n_indices (int) – number of indices to sample
Returns: number of indices to sample (truncated if necessary)
Return type: int
Raises: StopIteration – if enough batches sampled

_get_indices(n_indices)[source]¶

Actual Sampling

Parameters: n_indices (int) – number of indices to return

:raises StopIteration : If end of class indices is reached:

Returns: list of sampled indices
Return type: list

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset, **kwargs)[source]¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialzed sampler
Return type: AbstractSampler

StoppingPrevalenceSequentialSampler¶

class StoppingPrevalenceSequentialSampler(indices, shuffle_batch=True)[source]¶

Bases: delira.data_loading.sampler.abstract_sampler.AbstractSampler

Implements Per-Class Sequential sampling and ensures same number of samples per batch for each class; Stops if all samples of first class have been sampled

_check_batchsize(n_indices)[source]¶

Checks if batchsize is valid for all classes

Parameters: n_indices (int) – the number of samples to return
Returns: number of samples per class to return
Return type: dict

_get_indices(n_indices)[source]¶

Actual Sampling

Parameters: n_indices (int) – number of indices to return

:raises StopIteration : If end of class indices is reached for one class:

Returns: list of sampled indices
Return type: list

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset)[source]¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialzed sampler
Return type: AbstractSampler

WeightedRandomSampler¶

class WeightedRandomSampler(indices, weights=None)[source]¶

Bases: delira.data_loading.sampler.abstract_sampler.AbstractSampler

Implements Weighted Random Sampling

_check_batchsize(n_indices)¶

Checks if the batchsize is valid (and truncates batches if necessary). Will also raise StopIteration if enough batches sampled

Parameters: n_indices (int) – number of indices to sample
Returns: number of indices to sample (truncated if necessary)
Return type: int
Raises: StopIteration – if enough batches sampled

_get_indices(n_indices)[source]¶

Actual Sampling

Parameters

n_indices (int) – number of indices to return

Returns

list of sampled indices

Return type

list

Raises

StopIteration – If maximal number of samples is reached
ValueError – if weights or cum_weights don’t match the population

classmethod from_dataset(dataset: delira.data_loading.dataset.AbstractDataset, **kwargs)[source]¶

Classmethod to initialize the sampler from a given dataset

Parameters: dataset (AbstractDataset) – the given dataset
Returns: The initialzed sampler
Return type: AbstractSampler

IO¶

torch_load_checkpoint¶

load_checkpoint(file, **kwargs)[source]¶

Loads a saved model

Parameters

file (str) – filepath to a file containing a saved model
**kwargs – Additional keyword arguments (passed to torch.load) Especially “map_location” is important to change the device the state_dict should be loaded to

Returns

checkpoint state_dict

Return type

OrderedDict

torch_save_checkpoint¶

save_checkpoint(file: str, model=None, optimizers={}, epoch=None, **kwargs)[source]¶

Save model’s parameters

Parameters

file (str) – filepath the model should be saved to
model (AbstractNetwork or None) – the model which should be saved if None: empty dict will be saved as state dict
optimizers (dict) – dictionary containing all optimizers
epoch (int) – current epoch (will also be pickled)

tf_load_checkpoint¶

load_checkpoint(file: str, model=None)[source]¶

Loads a saved model

Parameters

file (str) – filepath to a file containing a saved model
model (TfNetwork) – the model which should be loaded

tf_save_checkpoint¶

save_checkpoint(file: str, model=None)[source]¶

Save model’s parameters contained in it’s graph

Parameters

file (str) – filepath the model should be saved to
model (TfNetwork) – the model which should be saved

Logging¶

This module handles the embedding of trixi’s loggers into the python logging module.

MultiStreamHandler¶

class MultiStreamHandler(*streams, level=0)[source]¶

Bases: logging.Handler

Logging Handler which accepts multiple streams and creates StreamHandlers

acquire()¶: Acquire the I/O thread lock.

addFilter(filter)¶: Add the specified filter to this handler.

close()¶

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.

createLock()¶: Acquire a thread lock for serializing access to the underlying I/O.

emit(record)[source]¶

logs the record entity to streams

Parameters: record (LogRecord) – record to log

filter(record)¶

Determine if a record is loggable by consulting all the filters.

The default is to allow the record to be logged; any filter can veto this and the record is then dropped. Returns a zero value if a record is to be dropped, else non-zero.

Changed in version 3.2: Allow filters to be just callables.

flush()¶

Ensure all logging output has been flushed.

This version does nothing and is intended to be implemented by subclasses.

format(record)¶

Format the specified record.

If a formatter is set, use it. Otherwise, use the default formatter for the module.

get_name()¶

handle(record)¶

Conditionally emit the specified logging record.

Emission depends on filters which may have been added to the handler. Wrap the actual emission of the record with acquisition/release of the I/O thread lock. Returns whether the filter passed the record for emission.

handleError(record)¶

Handle errors which occur during an emit() call.

This method should be called from handlers when an exception is encountered during an emit() call. If raiseExceptions is false, exceptions get silently ignored. This is what is mostly wanted for a logging system - most users will not care about errors in the logging system, they are more interested in application errors. You could, however, replace this with a custom handler if you wish. The record which was being processed is passed in to this method.

property name¶

release()¶: Release the I/O thread lock.

removeFilter(filter)¶: Remove the specified filter from this handler.

setFormatter(fmt)¶: Set the formatter for this handler.

setLevel(level)¶: Set the logging level of this handler. level must be an int or a str.

set_name(name)¶

TrixiHandler¶

class TrixiHandler(logging_cls, level=0, *args, **kwargs)[source]¶

Bases: logging.Handler

Handler to integrate the trixi loggers into the logging module

acquire()¶: Acquire the I/O thread lock.

addFilter(filter)¶: Add the specified filter to this handler.

close()¶

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.

createLock()¶: Acquire a thread lock for serializing access to the underlying I/O.

emit(record)[source]¶

logs the record entity to trixi loggers

Parameters: record (LogRecord) – record to log

filter(record)¶

Determine if a record is loggable by consulting all the filters.

The default is to allow the record to be logged; any filter can veto this and the record is then dropped. Returns a zero value if a record is to be dropped, else non-zero.

Changed in version 3.2: Allow filters to be just callables.

flush()¶

Ensure all logging output has been flushed.

This version does nothing and is intended to be implemented by subclasses.

format(record)¶

Format the specified record.

If a formatter is set, use it. Otherwise, use the default formatter for the module.

get_name()¶

handle(record)¶

Conditionally emit the specified logging record.

Emission depends on filters which may have been added to the handler. Wrap the actual emission of the record with acquisition/release of the I/O thread lock. Returns whether the filter passed the record for emission.

handleError(record)¶

Handle errors which occur during an emit() call.

This method should be called from handlers when an exception is encountered during an emit() call. If raiseExceptions is false, exceptions get silently ignored. This is what is mostly wanted for a logging system - most users will not care about errors in the logging system, they are more interested in application errors. You could, however, replace this with a custom handler if you wish. The record which was being processed is passed in to this method.

property name¶

release()¶: Release the I/O thread lock.

removeFilter(filter)¶: Remove the specified filter from this handler.

setFormatter(fmt)¶: Set the formatter for this handler.

setLevel(level)¶: Set the logging level of this handler. level must be an int or a str.

set_name(name)¶

Models¶

delira comes with it’s own model-structure tree - with AbstractNetwork at it’s root - and integrates PyTorch Models (AbstractPyTorchNetwork) deeply into the model structure. Tensorflow Integration is planned.

AbstractNetwork¶

class AbstractNetwork(type)[source]¶

Bases: object

Abstract class all networks should be derived from

_init_kwargs = {}¶

abstract static closure(model, data_dict: dict, optimizers: dict, losses={}, metrics={}, fold=0, **kwargs)[source]¶

Function which handles prediction from batch, logging, loss calculation and optimizer step :param model: model to forward data through :type model: AbstractNetwork :param data_dict: dictionary containing the data :type data_dict: dict :param optimizers: dictionary containing all optimizers to perform parameter update :type optimizers: dict :param losses: Functions or classes to calculate losses :type losses: dict :param metrics: Functions or classes to calculate other metrics :type metrics: dict :param fold: Current Fold in Crossvalidation (default: 0) :type fold: int :param kwargs: additional keyword arguments :type kwargs: dict

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as input dict losses)
dict – Arbitrary number of predictions

Raises

NotImplementedError – If not overwritten by subclass

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)[source]¶

Converts a numpy batch of data and labels to suitable datatype and pushes them to correct devices

Parameters

batch (dict) – dictionary containing the batch (must have keys ‘data’ and ‘label’
input_device – device for network inputs
output_device – device for network outputs

Returns

dictionary containing all necessary data in right format and type and on the correct device

Return type

dict

Raises

NotImplementedError – If not overwritten by subclass

AbstractPyTorchNetwork¶

class AbstractPyTorchNetwork(type)[source]¶

Bases: delira.models.abstract_network.AbstractNetwork, torch.nn.Module

Abstract Class for PyTorch Networks

See also

None, AbstractNetwork

_init_kwargs = {}¶

abstract static closure(model, data_dict: dict, optimizers: dict, losses={}, metrics={}, fold=0, **kwargs)¶

Function which handles prediction from batch, logging, loss calculation and optimizer step :param model: model to forward data through :type model: AbstractNetwork :param data_dict: dictionary containing the data :type data_dict: dict :param optimizers: dictionary containing all optimizers to perform parameter update :type optimizers: dict :param losses: Functions or classes to calculate losses :type losses: dict :param metrics: Functions or classes to calculate other metrics :type metrics: dict :param fold: Current Fold in Crossvalidation (default: 0) :type fold: int :param kwargs: additional keyword arguments :type kwargs: dict

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as input dict losses)
dict – Arbitrary number of predictions

Raises

NotImplementedError – If not overwritten by subclass

abstract forward(*inputs)[source]¶

Forward inputs through module (defines module behavior) :param inputs: inputs of arbitrary type and number :type inputs: list

Returns: result: module results of arbitrary type and number
Return type: Any

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)[source]¶

Helper Function to prepare Network Inputs and Labels (convert them to correct type and shape and push them to correct devices)

Parameters

batch (dict) – dictionary containing all the data
input_device (torch.device) – device for network inputs
output_device (torch.device) – device for network outputs

Returns

dictionary containing data in correct type and shape and on correct device

Return type

dict

AbstractTfNetwork¶

class AbstractTfNetwork(sess=tensorflow.Session, **kwargs)[source]¶

Bases: delira.models.abstract_network.AbstractNetwork

Abstract Class for Tf Networks

See also

AbstractNetwork

_add_losses(losses: dict)[source]¶

Add losses to the model graph

Parameters: losses (dict) – dictionary containing losses.

_add_optims(optims: dict)[source]¶

Add optimizers to the model graph

Parameters: optims (dict) – dictionary containing losses.

_init_kwargs = {}¶

abstract static closure(model, data_dict: dict, optimizers: dict, losses={}, metrics={}, fold=0, **kwargs)¶

Function which handles prediction from batch, logging, loss calculation and optimizer step :param model: model to forward data through :type model: AbstractNetwork :param data_dict: dictionary containing the data :type data_dict: dict :param optimizers: dictionary containing all optimizers to perform parameter update :type optimizers: dict :param losses: Functions or classes to calculate losses :type losses: dict :param metrics: Functions or classes to calculate other metrics :type metrics: dict :param fold: Current Fold in Crossvalidation (default: 0) :type fold: int :param kwargs: additional keyword arguments :type kwargs: dict

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as input dict losses)
dict – Arbitrary number of predictions

Raises

NotImplementedError – If not overwritten by subclass

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)¶

Converts a numpy batch of data and labels to suitable datatype and pushes them to correct devices

Parameters

batch (dict) – dictionary containing the batch (must have keys ‘data’ and ‘label’
input_device – device for network inputs
output_device – device for network outputs

Returns

dictionary containing all necessary data in right format and type and on the correct device

Return type

dict

Raises

NotImplementedError – If not overwritten by subclass

run(*args, **kwargs)[source]¶

Evaluates self.outputs_train or self.outputs_eval based on self.training

Parameters

*args – currently unused, exist for compatibility reasons
**kwargs – kwargs used to feed as self.inputs. Same keys as for self.inputs must be used

Returns

sames keys as outputs_train or outputs_eval, containing evaluated expressions as values

Return type

dict

Classification¶

ClassificationNetworkBasePyTorch¶

class ClassificationNetworkBasePyTorch(in_channels: int, n_outputs: int, **kwargs)[source]¶

Bases: delira.models.abstract_network.AbstractPyTorchNetwork

Implements basic classification with ResNet18

References

https://arxiv.org/abs/1512.03385

See also

AbstractPyTorchNetwork

static _build_model(in_channels: int, n_outputs: int, **kwargs)[source]¶

builds actual model (resnet 18)

Parameters

in_channels (int) – number of input channels
n_outputs (int) – number of outputs (usually same as number of classes)
**kwargs (dict) – additional keyword arguments

Returns

created model

Return type

torch.nn.Module

_init_kwargs = {}¶

static closure(model: delira.models.abstract_network.AbstractPyTorchNetwork, data_dict: dict, optimizers: dict, losses={}, metrics={}, fold=0, **kwargs)[source]¶

closure method to do a single backpropagation step

Parameters

model (ClassificationNetworkBasePyTorch) – trainable model
data_dict (dict) – dictionary containing the data
optimizers (dict) – dictionary of optimizers to optimize model’s parameters
losses (dict) – dict holding the losses to calculate errors (gradients from different losses will be accumulated)
metrics (dict) – dict holding the metrics to calculate
fold (int) – Current Fold in Crossvalidation (default: 0)
**kwargs – additional keyword arguments

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as input dict losses)
list – Arbitrary number of predictions as torch.Tensor

Raises

AssertionError – if optimizers or losses are empty or the optimizers are not specified

forward(input_batch: torch.Tensor)[source]¶

Forward input_batch through network

Parameters: input_batch (torch.Tensor) – batch to forward through network
Returns: Classification Result
Return type: torch.Tensor

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)[source]¶

Helper Function to prepare Network Inputs and Labels (convert them to correct type and shape and push them to correct devices)

Parameters

batch (dict) – dictionary containing all the data
input_device (torch.device) – device for network inputs
output_device (torch.device) – device for network outputs

Returns

dictionary containing data in correct type and shape and on correct device

Return type

dict

VGG3DClassificationNetworkPyTorch¶

class VGG3DClassificationNetworkPyTorch(in_channels: int, n_outputs: int, **kwargs)[source]¶

Bases: delira.models.classification.classification_network.ClassificationNetworkBasePyTorch

Exemplaric VGG Network for 3D Classification

Notes

The original network has been adjusted to fit for 3D data

References

https://arxiv.org/abs/1409.1556

See also

ClassificationNetworkBasePyTorch

static _build_model(in_channels: int, n_outputs: int, **kwargs)[source]¶

Helper Function to build the actual model

Parameters

in_channels (int) – number of input channels
n_outputs (int) – number of outputs
**kwargs – additional keyword arguments

Returns

ensembeled model

Return type

torch.nn.Module

_init_kwargs = {}¶

static closure(model: delira.models.abstract_network.AbstractPyTorchNetwork, data_dict: dict, optimizers: dict, losses={}, metrics={}, fold=0, **kwargs)¶

closure method to do a single backpropagation step

Parameters

model (ClassificationNetworkBasePyTorch) – trainable model
data_dict (dict) – dictionary containing the data
optimizers (dict) – dictionary of optimizers to optimize model’s parameters
losses (dict) – dict holding the losses to calculate errors (gradients from different losses will be accumulated)
metrics (dict) – dict holding the metrics to calculate
fold (int) – Current Fold in Crossvalidation (default: 0)
**kwargs – additional keyword arguments

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as input dict losses)
list – Arbitrary number of predictions as torch.Tensor

Raises

AssertionError – if optimizers or losses are empty or the optimizers are not specified

forward(input_batch: torch.Tensor)¶

Forward input_batch through network

Parameters: input_batch (torch.Tensor) – batch to forward through network
Returns: Classification Result
Return type: torch.Tensor

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)¶

Helper Function to prepare Network Inputs and Labels (convert them to correct type and shape and push them to correct devices)

Parameters

batch (dict) – dictionary containing all the data
input_device (torch.device) – device for network inputs
output_device (torch.device) – device for network outputs

Returns

dictionary containing data in correct type and shape and on correct device

Return type

dict

ClassificationNetworkBaseTf¶

class ClassificationNetworkBaseTf(in_channels: int, n_outputs: int, **kwargs)[source]¶

Bases: delira.models.abstract_network.AbstractTfNetwork

Implements basic classification with ResNet18

See also

AbstractTfNetwork

_add_losses(losses: dict)[source]¶

Adds losses to model that are to be used by optimizers or during evaluation

Parameters: losses (dict) – dictionary containing all losses. Individual losses are averaged

_add_optims(optims: dict)[source]¶

Adds optims to model that are to be used by optimizers or during training

Parameters: optim (dict) – dictionary containing all optimizers, optimizers should be of Type[tf.train.Optimizer]

static _build_model(n_outputs: int, **kwargs)[source]¶

builds actual model (resnet 18)

Parameters

n_outputs (int) – number of outputs (usually same as number of classes)
**kwargs – additional keyword arguments

Returns

created model

Return type

tf.keras.Model

_init_kwargs = {}¶

static closure(model: Type[delira.models.abstract_network.AbstractTfNetwork], data_dict: dict, metrics={}, fold=0, **kwargs)[source]¶

closure method to do a single prediction. This is followed by backpropagation or not based state of on model.train

Parameters

model (AbstractTfNetwork) – AbstractTfNetwork or its child-clases
data_dict (dict) – dictionary containing the data
metrics (dict) – dict holding the metrics to calculate
fold (int) – Current Fold in Crossvalidation (default: 0)
**kwargs – additional keyword arguments

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as those initially passed to model.init). Additionally, a total_loss key is added
dict – outputs of model.run

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)¶

Converts a numpy batch of data and labels to suitable datatype and pushes them to correct devices

Parameters

batch (dict) – dictionary containing the batch (must have keys ‘data’ and ‘label’
input_device – device for network inputs
output_device – device for network outputs

Returns

dictionary containing all necessary data in right format and type and on the correct device

Return type

dict

Raises

NotImplementedError – If not overwritten by subclass

run(*args, **kwargs)¶

Evaluates self.outputs_train or self.outputs_eval based on self.training

Parameters

*args – currently unused, exist for compatibility reasons
**kwargs – kwargs used to feed as self.inputs. Same keys as for self.inputs must be used

Returns

sames keys as outputs_train or outputs_eval, containing evaluated expressions as values

Return type

dict

Generative Adversarial Networks¶

GenerativeAdversarialNetworkBasePyTorch¶

class GenerativeAdversarialNetworkBasePyTorch(n_channels, noise_length, **kwargs)[source]¶

Bases: delira.models.abstract_network.AbstractPyTorchNetwork

Implementation of Vanilla DC-GAN to create 64x64 pixel images

Notes

The fully connected part in the discriminator has been replaced with an equivalent convolutional part

References

https://arxiv.org/abs/1511.06434

See also

AbstractPyTorchNetwork

static _build_models(in_channels, noise_length, **kwargs)[source]¶

Builds actual generator and discriminator models

Parameters

in_channels (int) – number of channels for generated images by generator and inputs of discriminator
noise_length (int) – length of noise vector (generator input)
**kwargs – additional keyword arguments

Returns

torch.nn.Sequential – generator
torch.nn.Sequential – discriminator

_init_kwargs = {}¶

static closure(model, data_dict: dict, optimizers: dict, losses={}, metrics={}, fold=0, **kwargs)[source]¶

closure method to do a single backpropagation step

Parameters

model (ClassificationNetworkBase) – trainable model
data_dict (dict) – dictionary containing data
optimizers (dict) – dictionary of optimizers to optimize model’s parameters
losses (dict) – dict holding the losses to calculate errors (gradients from different losses will be accumulated)
metrics (dict) – dict holding the metrics to calculate
fold (int) – Current Fold in Crossvalidation (default: 0)
kwargs (dict) – additional keyword arguments

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as input dict losses)
list – Arbitrary number of predictions as torch.Tensor

Raises

AssertionError – if optimizers or losses are empty or the optimizers are not specified

forward(real_image_batch)[source]¶

Create fake images by feeding noise through generator and feed results and real images through discriminator

Parameters

real_image_batch (torch.Tensor) – batch of real images

Returns

torch.Tensor – Generated fake images
torch.Tensor – Discriminator prediction of fake images
torch.Tensor – Discriminator prediction of real images

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)¶

Helper Function to prepare Network Inputs and Labels (convert them to correct type and shape and push them to correct devices)

Parameters

batch (dict) – dictionary containing all the data
input_device (torch.device) – device for network inputs
output_device (torch.device) – device for network outputs

Returns

dictionary containing data in correct type and shape and on correct device

Return type

dict

Segmentation¶

UNet2dPyTorch¶

class UNet2dPyTorch(num_classes, in_channels=1, depth=5, start_filts=64, up_mode='transpose', merge_mode='concat')[source]¶

Bases: delira.models.abstract_network.AbstractPyTorchNetwork

The UNet2dPyTorch is a convolutional encoder-decoder neural network. Contextual spatial information (from the decoding, expansive pathway) about an input tensor is merged with information representing the localization of details (from the encoding, compressive pathway).

Notes

Differences to the original paper:

padding is used in 3x3 convolutions to prevent loss of border
pixels

merging outputs does not require cropping due to (1)

residual connections can be used by specifying
merge_mode='add'

if non-parametric upsampling is used in the decoder pathway (
specified by upmode=’upsample’), then an additional 1x1 2d convolution occurs after upsampling to reduce channel dimensionality by a factor of 2. This channel halving happens with the convolution in the tranpose convolution (specified by upmode='transpose')

References

https://arxiv.org/abs/1505.04597

See also

UNet3dPyTorch

_build_model(num_classes, in_channels=3, depth=5, start_filts=64)[source]¶

Builds the actual model

Parameters

num_classes (int) – number of output classes
in_channels (int) – number of channels for the input tensor (default: 1)
depth (int) – number of MaxPools in the U-Net (default: 5)
start_filts (int) – number of convolutional filters for the first conv (affects all other conv-filter numbers too; default: 64)

Notes

The Helper functions and classes are defined within this function because delira once offered a possibility to save the source code along the weights to completely recover the network without needing a manually created network instance and these helper functions had to be saved too.

_init_kwargs = {}¶

static closure(model, data_dict: dict, optimizers: dict, losses={}, metrics={}, fold=0, **kwargs)[source]¶

closure method to do a single backpropagation step

Parameters

model (ClassificationNetworkBasePyTorch) – trainable model
data_dict (dict) – dictionary containing the data
optimizers (dict) – dictionary of optimizers to optimize model’s parameters
losses (dict) – dict holding the losses to calculate errors (gradients from different losses will be accumulated)
metrics (dict) – dict holding the metrics to calculate
fold (int) – Current Fold in Crossvalidation (default: 0)
**kwargs – additional keyword arguments

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as input dict losses)
list – Arbitrary number of predictions as torch.Tensor

Raises

AssertionError – if optimizers or losses are empty or the optimizers are not specified

forward(x)[source]¶

Feed tensor through network

Parameters: x (torch.Tensor) –
Returns: Prediction
Return type: torch.Tensor

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)[source]¶

Helper Function to prepare Network Inputs and Labels (convert them to correct type and shape and push them to correct devices)

Parameters

batch (dict) – dictionary containing all the data
input_device (torch.device) – device for network inputs
output_device (torch.device) – device for network outputs

Returns

dictionary containing data in correct type and shape and on correct device

Return type

dict

reset_params()[source]¶: Initialize all parameters

static weight_init(m)[source]¶

Initializes weights with xavier_normal and bias with zeros

Parameters: m (torch.nn.Module) – module to initialize

UNet3dPyTorch¶

class UNet3dPyTorch(num_classes, in_channels=3, depth=5, start_filts=64, up_mode='transpose', merge_mode='concat')[source]¶

Bases: delira.models.abstract_network.AbstractPyTorchNetwork

The UNet3dPyTorch is a convolutional encoder-decoder neural network. Contextual spatial information (from the decoding, expansive pathway) about an input tensor is merged with information representing the localization of details (from the encoding, compressive pathway).

Notes

Differences to the original paper:

Working on 3D data instead of 2D slices
padding is used in 3x3x3 convolutions to prevent loss of border
pixels
merging outputs does not require cropping due to (1)
residual connections can be used by specifying
merge_mode='add'
if non-parametric upsampling is used in the decoder pathway (
specified by upmode=’upsample’), then an additional 1x1x1 3d convolution occurs after upsampling to reduce channel dimensionality by a factor of 2. This channel halving happens with the convolution in the tranpose convolution (specified by upmode='transpose')

References

https://arxiv.org/abs/1505.04597

See also

UNet2dPyTorch

_build_model(num_classes, in_channels=3, depth=5, start_filts=64)[source]¶

Builds the actual model

Parameters

num_classes (int) – number of output classes
in_channels (int) – number of channels for the input tensor (default: 1)
depth (int) – number of MaxPools in the U-Net (default: 5)
start_filts (int) – number of convolutional filters for the first conv (affects all other conv-filter numbers too; default: 64)

Notes

The Helper functions and classes are defined within this function because delira once offered a possibility to save the source code along the weights to completely recover the network without needing a manually created network instance and these helper functions had to be saved too.

_init_kwargs = {}¶

static closure(model, data_dict: dict, optimizers: dict, losses={}, metrics={}, fold=0, **kwargs)[source]¶

closure method to do a single backpropagation step

Parameters

model (ClassificationNetworkBasePyTorch) – trainable model
data_dict (dict) – dictionary containing the data
optimizers (dict) – dictionary of optimizers to optimize model’s parameters
losses (dict) – dict holding the losses to calculate errors (gradients from different losses will be accumulated)
metrics (dict) – dict holding the metrics to calculate
fold (int) – Current Fold in Crossvalidation (default: 0)
**kwargs – additional keyword arguments

Returns

dict – Metric values (with same keys as input dict metrics)
dict – Loss values (with same keys as input dict losses)
list – Arbitrary number of predictions as torch.Tensor

Raises

AssertionError – if optimizers or losses are empty or the optimizers are not specified

forward(x)[source]¶

Feed tensor through network

Parameters: x (torch.Tensor) –
Returns: Prediction
Return type: torch.Tensor

property init_kwargs¶

Returns all arguments registered as init kwargs

Returns: init kwargs
Return type: dict

static prepare_batch(batch: dict, input_device, output_device)[source]¶

Helper Function to prepare Network Inputs and Labels (convert them to correct type and shape and push them to correct devices)

Parameters

batch (dict) – dictionary containing all the data
input_device (torch.device) – device for network inputs
output_device (torch.device) – device for network outputs

Returns

dictionary containing data in correct type and shape and on correct device

Return type

dict

reset_params()[source]¶: Initialize all parameters

static weight_init(m)[source]¶

Initializes weights with xavier_normal and bias with zeros

Parameters: m (torch.nn.Module) – module to initialize

Training¶

The training subpackage implements Callbacks, a class for Hyperparameters, training routines and wrapping experiments.

Parameters¶

class Parameters(fixed_params={'model': {}, 'training': {}}, variable_params={'model': {}, 'training': {}})[source]¶

Bases: delira.utils.config.LookupConfig

Class Containing all variable and fixed parameters for training and model instantiation

See also

trixi.util.Config

property hierarchy¶

Returns the current hierarchy

Returns: current hierarchy
Return type: str

nested_get(key, *args, **kwargs)¶

Returns all occurances of key in self and subdicts

Parameters

key (str) – the key to search for
*args – positional arguments to provide default value
**kwargs – keyword arguments to provide default value

Raises

KeyError – Multiple Values are found for key (unclear which value should be returned) OR No Value was found for key and no default value was given

Returns

value corresponding to key (or default if value was not found)

Return type

Any

permute_hierarchy()[source]¶

switches hierarchy

Returns: the class with a permuted hierarchy
Return type: Parameters
Raises: AttributeError – if no valid hierarchy is found

permute_to_hierarchy(hierarchy: str)[source]¶

Permute hierarchy to match the specified hierarchy

Parameters: hierarchy (str) – target hierarchy
Raises: ValueError – Specified hierarchy is invalid
Returns: parameters with proper hierarchy
Return type: Parameters

permute_training_on_top()[source]¶

permutes hierarchy in a way that the training-model hierarchy is on top

Returns: Parameters with permuted hierarchy
Return type: Parameters

permute_variability_on_top()[source]¶

permutes hierarchy in a way that the training-model hierarchy is on top

Returns: Parameters with permuted hierarchy
Return type: Parameters

save(filepath: str)[source]¶

Saves class to given filepath (YAML + Pickle)

Parameters: filepath (str) – file to save data to

property training_on_top¶

Return whether the training hierarchy is on top

Returns: whether training is on top
Return type: bool

update(dict_like, deep=False, ignore=None, allow_dict_overwrite=True)[source]¶

Update entries in the Parameters

Parameters

dict_like (dict) – Update source
deep (bool) – Make deep copies of all references in the source.
ignore (Iterable) – Iterable of keys to ignore in update
allow_dict_overwrite (bool) – Allow overwriting with dict. Regular dicts only update on the highest level while we recurse and merge Configs. This flag decides whether it is possible to overwrite a ‘regular’ value with a dict/Config at lower levels. See examples for an illustration of the difference
Examples –
--------- –
following illustrates the update behaviour if (The) –

:param : :type : obj:allow_dict_overwrite is active. If it isn’t, an AttributeError :param would be raised, originating from trying to update “string”::

config1 = Config(config={
    "lvl0": {
        "lvl1": "string",
        "something": "else"
    }
})

config2 = Config(config={
    "lvl0": {
        "lvl1": {
            "lvl2": "string"
        }
    }
})

config1.update(config2, allow_dict_overwrite=True)

>>>config1
{
    "lvl0": {
        "lvl1": {
            "lvl2": "string"
        },
        "something": "else"
    }
}

property variability_on_top¶

Return whether the variability is on top

Returns: whether variability is on top
Return type: bool

NetworkTrainer¶

The network trainer implements the actual training routine and can be subclassed: for special routines.

BaseNetworkTrainer¶

class BaseNetworkTrainer(network: delira.models.abstract_network.AbstractNetwork, save_path: str, losses: dict, optimizer_cls: type, optimizer_params: dict, train_metrics: dict, val_metrics: dict, lr_scheduler_cls: type, lr_scheduler_params: dict, gpu_ids: List[int], save_freq: int, optim_fn, key_mapping: dict, logging_type: str, logging_kwargs: dict, fold: int, callbacks: List[delira.training.callbacks.abstract_callback.AbstractCallback], start_epoch=1, metric_keys=None, convert_batch_to_npy_fn=<function BaseNetworkTrainer.<lambda>>, val_freq=1, **kwargs)[source]¶

Bases: delira.training.predictor.Predictor

Defines a Base API and basic functions for Network Trainers

_BaseNetworkTrainer__KEYS_TO_GUARD = ['use_gpu', 'input_device', 'output_device', '_callbacks']¶

_Predictor__KEYS_TO_GUARD = []¶

static _Predictor__concatenate_dict_items(dict_like: dict)¶

Function to recursively concatenate dict-items

Parameters: dict_like (dict) – the (nested) dict, whoose items should be concatenated

static _Predictor__convert_dict(old_dict, new_dict)¶

Function to recursively convert dicts

Parameters

old_dict (dict) – the old nested dict
new_dict (dict) – the new nested dict

Returns

the updated new nested dict

Return type

dict

_at_epoch_begin(metrics_val, val_score_key, epoch, num_epochs, **kwargs)[source]¶

Defines behaviour at beginning of each epoch: Executes all callbacks’s at_epoch_begin method

Parameters

metrics_val (dict) – validation metrics
val_score_key (str) – validation score key
epoch (int) – current epoch
num_epochs (int) – total number of epochs
**kwargs – keyword arguments

_at_epoch_end(metrics_val, val_score_key, epoch, is_best, **kwargs)[source]¶

Defines behaviour at beginning of each epoch: Executes all callbacks’s at_epoch_end method and saves current state if necessary

Parameters

metrics_val (dict) – validation metrics
val_score_key (str) – validation score key
epoch (int) – current epoch
num_epochs (int) – total number of epochs
**kwargs – keyword arguments

_at_training_begin(*args, **kwargs)[source]¶

Defines the behaviour at beginnig of the training

Parameters

*args – positional arguments
**kwargs – keyword arguments

Raises

NotImplementedError – If not overwritten by subclass

_at_training_end(*args, **kwargs)[source]¶

Defines the behaviour at the end of the training

Parameters

*args – positional arguments
**kwargs – keyword arguments

Raises

NotImplementedError – If not overwritten by subclass

static _is_better_val_scores(old_val_score, new_val_score, mode='highest')[source]¶

Check whether the new val score is better than the old one with respect to the optimization goal

Parameters

old_val_score – old validation score
new_val_score – new validation score
mode (str) – String to specify whether a higher or lower validation score is optimal; must be in [‘highest’, ‘lowest’]

Returns

True if new score is better, False otherwise

Return type

bool

_reinitialize_logging(logging_type, logging_kwargs: dict)[source]¶

static _search_for_prev_state(path, extensions=[])[source]¶

Helper function to search in a given path for previous epoch states (indicated by extensions)

Parameters

path (str) – the path to search in
extensions (list) – list of strings containing valid file extensions for checkpoint files

Returns

str – the file containing the latest checkpoint (if available)
None – if no latst checkpoint was found
int – the latest epoch (1 if no checkpoint was found)

_setup(network, lr_scheduler_cls, lr_scheduler_params, gpu_ids, key_mapping, convert_batch_to_npy_fn, prepare_batch_fn)[source]¶

Parameters

network (AbstractNetwork) – the network to predict from
key_mapping (dict) – a dictionary containing the mapping from the data_dict to the actual model’s inputs. E.g. if a model accepts one input named ‘x’ and the data_dict contains one entry named ‘data’ this argument would have to be {'x': 'data'}
convert_batch_to_npy_fn (type) – a callable function to convert tensors in positional and keyword arguments to numpy
prepare_batch_fn (type) – function converting a batch-tensor to the framework specific tensor-type and pushing it to correct device, default: identity function

_train_single_epoch(batchgen: delira.data_loading.data_manager.Augmenter, epoch, verbose=False)[source]¶

Trains the network a single epoch

Parameters

batchgen (Augmenter) – Generator yielding the training batches
epoch (int) – current epoch

_update_state(new_state)[source]¶

Update the state from a given new state

Parameters: new_state (dict) – new state to update internal state from
Returns: the trainer with a modified state
Return type: BaseNetworkTrainer

static calc_metrics(batch: delira.utils.config.LookupConfig, metrics={}, metric_keys=None)¶

Compute metrics

Parameters

batch (LookupConfig) – dictionary containing the whole batch (including predictions)
metrics (dict) – dict with metrics
metric_keys (dict) – dict of tuples which contains hashables for specifying the items to use for calculating the respective metric. If not specified for a metric, the keys “pred” and “label” are used per default

Returns

dict with metric results

Return type

dict

property fold¶

Get current fold

Returns: current fold
Return type: int

static load_state(file_name, *args, **kwargs)[source]¶

Loads the new state from file

Parameters

file_name (str) – the file to load the state from
*args – positional arguments
**kwargs (keyword arguments) –

Returns

new state

Return type

dict

predict(data: dict, **kwargs)¶

Predict single batch Returns the predictions corresponding to the given data obtained by the model

Parameters

data (dict) – batch dictionary
**kwargs – keyword arguments(directly passed to prepare_batch)

Returns

predicted data

Return type

dict

predict_data_mgr(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator without explicitly caching anything

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all predictions of the current batch
dict – a dictionary containing all metrics of the current batch

predict_data_mgr_cache(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, cache_preds=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches all predictions and metrics (yields them in dicts)

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all validation metrics (maybe empty)
dict – a dictionary containing all predictions; If cache_preds=True

Warning

Since this function caches all metrics and may additionally cache all predictions (based on the argument cache_preds), this may result in huge memory consumption. If you are running out of memory, please have a look at Predictor.predict_data_mgr_cache_metrics_only() or Predictor.predict_data_mgr() or consider setting cache_preds to False (if not done already)

predict_data_mgr_cache_all(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches all predictions and metrics (yields them in dicts)

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all predictions;
dict – a dictionary containing all validation metrics (maybe empty)

Warning

Since this function caches all predictions and metrics, this may result in huge memory consumption. If you are running out of memory, please have a look at Predictor.predict_data_mgr_cache_metrics_only() or Predictor.predict_data_mgr()

predict_data_mgr_cache_metrics_only(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches the metrics

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all validation metrics (maybe empty)

Notes

This function stores each prediction temporarily for metric calculation; This results in a (typically) way lower memory consumption than Predictor.predict_data_mgr_cache_all(), but still caches the metrics. If this is not desired, it is recommended to use Predictor.predict_data_mgr() and iterate over the generator as this only produces per-batch metrics and predictions and does not cache anything by default

register_callback(callback: delira.training.callbacks.abstract_callback.AbstractCallback)[source]¶

Register Callback to Trainer

Parameters: callback (AbstractCallback) – the callback to register
Raises: AssertionError – callback is not an instance of AbstractCallback and has not both methods [‘at_epoch_begin’, ‘at_epoch_end’]

save_state(file_name, *args, **kwargs)[source]¶

saves the current state

Parameters

file_name (str) – filename to save the state to
*args – positional arguments
**kwargs – keyword arguments

train(num_epochs, datamgr_train, datamgr_valid=None, val_score_key=None, val_score_mode='highest', reduce_mode='mean', verbose=True)[source]¶

Defines a routine to train a specified number of epochs

Parameters

num_epochs (int) – number of epochs to train
datamgr_train (DataManager) – the datamanager holding the train data
datamgr_valid (DataManager) – the datamanager holding the validation data (default: None)
val_score_key (str) – the key specifying which metric to use for validation (default: None)
val_score_mode (str) – key specifying what kind of validation score is best
reduce_mode (str) – ‘mean’,’sum’,’first_only’
verbose (bool) – whether to show progress bars or not

Raises

NotImplementedError – If not overwritten by subclass

update_state(file_name, *args, **kwargs)[source]¶

Update internal state from a loaded state

Parameters

file_name (str) – file containing the new state to load
*args – positional arguments
**kwargs – keyword arguments

Returns

the trainer with a modified state

Return type

BaseNetworkTrainer

PyTorchNetworkTrainer¶

class PyTorchNetworkTrainer(network: delira.models.abstract_network.AbstractPyTorchNetwork, save_path: str, key_mapping, losses=None, optimizer_cls=None, optimizer_params={}, train_metrics={}, val_metrics={}, lr_scheduler_cls=None, lr_scheduler_params={}, gpu_ids=[], save_freq=1, optim_fn=<function create_optims_default_pytorch>, logging_type='tensorboardx', logging_kwargs={}, fold=0, callbacks=[], start_epoch=1, metric_keys=None, convert_batch_to_npy_fn=<function convert_torch_tensor_to_npy>, mixed_precision=False, mixed_precision_kwargs={'allow_banned': False, 'enable_caching': True, 'verbose': False}, criterions=None, val_freq=1, **kwargs)[source]¶

Bases: delira.training.base_trainer.BaseNetworkTrainer

Train and Validate a Network

See also

AbstractNetwork

_BaseNetworkTrainer__KEYS_TO_GUARD = ['use_gpu', 'input_device', 'output_device', '_callbacks']¶

_Predictor__KEYS_TO_GUARD = []¶

static _Predictor__concatenate_dict_items(dict_like: dict)¶

Function to recursively concatenate dict-items

Parameters: dict_like (dict) – the (nested) dict, whoose items should be concatenated

static _Predictor__convert_dict(old_dict, new_dict)¶

Function to recursively convert dicts

Parameters

old_dict (dict) – the old nested dict
new_dict (dict) – the new nested dict

Returns

the updated new nested dict

Return type

dict

_at_epoch_begin(metrics_val, val_score_key, epoch, num_epochs, **kwargs)¶

Defines behaviour at beginning of each epoch: Executes all callbacks’s at_epoch_begin method

Parameters

metrics_val (dict) – validation metrics
val_score_key (str) – validation score key
epoch (int) – current epoch
num_epochs (int) – total number of epochs
**kwargs – keyword arguments

_at_epoch_end(metrics_val, val_score_key, epoch, is_best, **kwargs)[source]¶

Defines behaviour at beginning of each epoch: Executes all callbacks’s at_epoch_end method and saves current state if necessary

Parameters

metrics_val (dict) – validation metrics
val_score_key (str) – validation score key
epoch (int) – current epoch
num_epochs (int) – total number of epochs
is_best (bool) – whether current model is best one so far
**kwargs – keyword arguments

_at_training_begin(*args, **kwargs)[source]¶

Defines behaviour at beginning of training

Parameters

*args – positional arguments
**kwargs – keyword arguments

_at_training_end()[source]¶

Defines Behaviour at end of training: Loads best model if available

Returns: best network
Return type: AbstractPyTorchNetwork

static _is_better_val_scores(old_val_score, new_val_score, mode='highest')¶

Check whether the new val score is better than the old one with respect to the optimization goal

Parameters

old_val_score – old validation score
new_val_score – new validation score
mode (str) – String to specify whether a higher or lower validation score is optimal; must be in [‘highest’, ‘lowest’]

Returns

True if new score is better, False otherwise

Return type

bool

_reinitialize_logging(logging_type, logging_kwargs: dict)¶

static _search_for_prev_state(path, extensions=[])¶

Helper function to search in a given path for previous epoch states (indicated by extensions)

Parameters

path (str) – the path to search in
extensions (list) – list of strings containing valid file extensions for checkpoint files

Returns

str – the file containing the latest checkpoint (if available)
None – if no latst checkpoint was found
int – the latest epoch (1 if no checkpoint was found)

_setup(network, optim_fn, optimizer_cls, optimizer_params, lr_scheduler_cls, lr_scheduler_params, gpu_ids, key_mapping, convert_batch_to_npy_fn, mixed_precision, mixed_precision_kwargs)[source]¶

Defines the Trainers Setup

Parameters

network (AbstractPyTorchNetwork) – the network to train
optim_fn (function) – creates a dictionary containing all necessary optimizers
optimizer_cls (subclass of torch.optim.Optimizer) – optimizer class implementing the optimization algorithm of choice
optimizer_params (dict) –
lr_scheduler_cls (Any) – learning rate schedule class: must implement step() method
lr_scheduler_params (dict) – keyword arguments passed to lr scheduler during construction
gpu_ids (list) – list containing ids of GPUs to use; if empty: use cpu instead
convert_batch_to_npy_fn (type) – function converting a batch-tensor to numpy
mixed_precision (bool) – whether to use mixed precision or not (False per default)
mixed_precision_kwargs (dict) – additional keyword arguments for mixed precision

_train_single_epoch(batchgen: batchgenerators.dataloading.MultiThreadedAugmenter, epoch, verbose=False)[source]¶

Trains the network a single epoch

Parameters

batchgen (MultiThreadedAugmenter) – Generator yielding the training batches
epoch (int) – current epoch

_update_state(new_state)[source]¶

Update the state from a given new state

Parameters: new_state (dict) – new state to update internal state from
Returns: the trainer with a modified state
Return type: PyTorchNetworkTrainer

static calc_metrics(batch: delira.utils.config.LookupConfig, metrics={}, metric_keys=None)¶

Compute metrics

Parameters

batch (LookupConfig) – dictionary containing the whole batch (including predictions)
metrics (dict) – dict with metrics
metric_keys (dict) – dict of tuples which contains hashables for specifying the items to use for calculating the respective metric. If not specified for a metric, the keys “pred” and “label” are used per default

Returns

dict with metric results

Return type

dict

property fold¶

Get current fold

Returns: current fold
Return type: int

static load_state(file_name, **kwargs)[source]¶

Loads the new state from file via delira.io.torch.load_checkpoint()

Parameters

file_name (str) – the file to load the state from
**kwargs (keyword arguments) –

Returns

new state

Return type

dict

predict(data: dict, **kwargs)¶

Predict single batch Returns the predictions corresponding to the given data obtained by the model

Parameters

data (dict) – batch dictionary
**kwargs – keyword arguments(directly passed to prepare_batch)

Returns

predicted data

Return type

dict

predict_data_mgr(datamgr, batchsize=None, metrics={}, metric_keys={}, verbose=False, **kwargs)[source]¶

Defines a routine to predict data obtained from a batchgenerator

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
**kwargs – additional keyword arguments

Returns

dict – predictions
dict – calculated metrics

predict_data_mgr_cache(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, cache_preds=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches all predictions and metrics (yields them in dicts)

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all validation metrics (maybe empty)
dict – a dictionary containing all predictions; If cache_preds=True

Warning

Since this function caches all metrics and may additionally cache all predictions (based on the argument cache_preds), this may result in huge memory consumption. If you are running out of memory, please have a look at Predictor.predict_data_mgr_cache_metrics_only() or Predictor.predict_data_mgr() or consider setting cache_preds to False (if not done already)

predict_data_mgr_cache_all(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches all predictions and metrics (yields them in dicts)

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all predictions;
dict – a dictionary containing all validation metrics (maybe empty)

Warning

Since this function caches all predictions and metrics, this may result in huge memory consumption. If you are running out of memory, please have a look at Predictor.predict_data_mgr_cache_metrics_only() or Predictor.predict_data_mgr()

predict_data_mgr_cache_metrics_only(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches the metrics

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all validation metrics (maybe empty)

Notes

This function stores each prediction temporarily for metric calculation; This results in a (typically) way lower memory consumption than Predictor.predict_data_mgr_cache_all(), but still caches the metrics. If this is not desired, it is recommended to use Predictor.predict_data_mgr() and iterate over the generator as this only produces per-batch metrics and predictions and does not cache anything by default

register_callback(callback: delira.training.callbacks.abstract_callback.AbstractCallback)¶

Register Callback to Trainer

Parameters: callback (AbstractCallback) – the callback to register
Raises: AssertionError – callback is not an instance of AbstractCallback and has not both methods [‘at_epoch_begin’, ‘at_epoch_end’]

save_state(file_name, epoch, **kwargs)[source]¶

saves the current state via delira.io.torch.save_checkpoint()

Parameters

file_name (str) – filename to save the state to
epoch (int) – current epoch (will be saved for mapping back)
*args – positional arguments
**kwargs – keyword arguments

train(num_epochs, datamgr_train, datamgr_valid=None, val_score_key=None, val_score_mode='highest', reduce_mode='mean', verbose=True)¶

Defines a routine to train a specified number of epochs

Parameters

num_epochs (int) – number of epochs to train
datamgr_train (DataManager) – the datamanager holding the train data
datamgr_valid (DataManager) – the datamanager holding the validation data (default: None)
val_score_key (str) – the key specifying which metric to use for validation (default: None)
val_score_mode (str) – key specifying what kind of validation score is best
reduce_mode (str) – ‘mean’,’sum’,’first_only’
verbose (bool) – whether to show progress bars or not

Raises

NotImplementedError – If not overwritten by subclass

update_state(file_name, *args, **kwargs)[source]¶

Update internal state from a loaded state

Parameters

file_name (str) – file containing the new state to load
*args – positional arguments
**kwargs – keyword arguments

Returns

the trainer with a modified state

Return type

BaseNetworkTrainer

TfNetworkTrainer¶

class TfNetworkTrainer(network, save_path, key_mapping, losses: dict, optimizer_cls, optimizer_params={}, train_metrics={}, val_metrics={}, lr_scheduler_cls=None, lr_scheduler_params={}, gpu_ids=[], save_freq=1, optim_fn=<function create_optims_default_tf>, logging_type='tensorboardx', logging_kwargs={}, fold=0, callbacks=[], start_epoch=1, metric_keys=None, convert_batch_to_npy_fn=<function convert_tf_tensor_to_npy>, val_freq=1, **kwargs)[source]¶

Bases: delira.training.base_trainer.BaseNetworkTrainer

Train and Validate a Network

See also

AbstractNetwork

_BaseNetworkTrainer__KEYS_TO_GUARD = ['use_gpu', 'input_device', 'output_device', '_callbacks']¶

_Predictor__KEYS_TO_GUARD = []¶

static _Predictor__concatenate_dict_items(dict_like: dict)¶

Function to recursively concatenate dict-items

Parameters: dict_like (dict) – the (nested) dict, whoose items should be concatenated

static _Predictor__convert_dict(old_dict, new_dict)¶

Function to recursively convert dicts

Parameters

old_dict (dict) – the old nested dict
new_dict (dict) – the new nested dict

Returns

the updated new nested dict

Return type

dict

_at_epoch_begin(metrics_val, val_score_key, epoch, num_epochs, **kwargs)¶

Defines behaviour at beginning of each epoch: Executes all callbacks’s at_epoch_begin method

Parameters

metrics_val (dict) – validation metrics
val_score_key (str) – validation score key
epoch (int) – current epoch
num_epochs (int) – total number of epochs
**kwargs – keyword arguments

_at_epoch_end(metrics_val, val_score_key, epoch, is_best, **kwargs)¶

Defines behaviour at beginning of each epoch: Executes all callbacks’s at_epoch_end method and saves current state if necessary

Parameters

metrics_val (dict) – validation metrics
val_score_key (str) – validation score key
epoch (int) – current epoch
num_epochs (int) – total number of epochs
**kwargs – keyword arguments

_at_training_begin(*args, **kwargs)¶

Defines the behaviour at beginnig of the training

Parameters

*args – positional arguments
**kwargs – keyword arguments

Raises

NotImplementedError – If not overwritten by subclass

_at_training_end()[source]¶

Defines Behaviour at end of training: Loads best model if available

Returns: best network
Return type: AbstractTfNetwork

static _is_better_val_scores(old_val_score, new_val_score, mode='highest')¶

Check whether the new val score is better than the old one with respect to the optimization goal

Parameters

old_val_score – old validation score
new_val_score – new validation score
mode (str) – String to specify whether a higher or lower validation score is optimal; must be in [‘highest’, ‘lowest’]

Returns

True if new score is better, False otherwise

Return type

bool

_reinitialize_logging(logging_type, logging_kwargs: dict)¶

static _search_for_prev_state(path, extensions=[])¶

Helper function to search in a given path for previous epoch states (indicated by extensions)

Parameters

path (str) – the path to search in
extensions (list) – list of strings containing valid file extensions for checkpoint files

Returns

str – the file containing the latest checkpoint (if available)
None – if no latst checkpoint was found
int – the latest epoch (1 if no checkpoint was found)

_setup(network, optim_fn, optimizer_cls, optimizer_params, lr_scheduler_cls, lr_scheduler_params, key_mapping, convert_batch_to_npy_fn, gpu_ids)[source]¶

Defines the Trainers Setup

Parameters

network (instance of :class: AbstractTfNetwork) – the network to train
optim_fn (function) – creates a dictionary containing all necessary optimizers
optimizer_cls (subclass of tf.train.Optimizer) – optimizer class implementing the optimization algorithm of choice
optimizer_params (dict) –
lr_scheduler_cls (Any) – learning rate schedule class: must implement step() method
lr_scheduler_params (dict) – keyword arguments passed to lr scheduler during construction
convert_batch_to_npy_fn (type, optional) – function converting a batch-tensor to numpy, per default this is the identity function
gpu_ids (list) – list containing ids of GPUs to use; if empty: use cpu instead

_train_single_epoch(batchgen: batchgenerators.dataloading.MultiThreadedAugmenter, epoch, verbose=False)[source]¶

Trains the network a single epoch

Parameters

batchgen (MultiThreadedAugmenter) – Generator yielding the training batches
epoch (int) – current epoch

_update_state(new_state)¶

Update the state from a given new state

Parameters: new_state (dict) – new state to update internal state from
Returns: the trainer with a modified state
Return type: BaseNetworkTrainer

static calc_metrics(batch: delira.utils.config.LookupConfig, metrics={}, metric_keys=None)¶

Compute metrics

Parameters

batch (LookupConfig) – dictionary containing the whole batch (including predictions)
metrics (dict) – dict with metrics
metric_keys (dict) – dict of tuples which contains hashables for specifying the items to use for calculating the respective metric. If not specified for a metric, the keys “pred” and “label” are used per default

Returns

dict with metric results

Return type

dict

property fold¶

Get current fold

Returns: current fold
Return type: int

load_state(file_name, *args, **kwargs)[source]¶

Loads the new state from file via delira.io.tf.load_checkpoint()

Parameters: file_name (str) – the file to load the state from

predict(data: dict, **kwargs)¶

Predict single batch Returns the predictions corresponding to the given data obtained by the model

Parameters

data (dict) – batch dictionary
**kwargs – keyword arguments(directly passed to prepare_batch)

Returns

predicted data

Return type

dict

predict_data_mgr(datamgr, batch_size=None, metrics={}, metric_keys={}, verbose=False, **kwargs)[source]¶

Defines a routine to predict data obtained from a batchgenerator

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
**kwargs – additional keword arguments

predict_data_mgr_cache(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, cache_preds=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches all predictions and metrics (yields them in dicts)

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all validation metrics (maybe empty)
dict – a dictionary containing all predictions; If cache_preds=True

Warning

Since this function caches all metrics and may additionally cache all predictions (based on the argument cache_preds), this may result in huge memory consumption. If you are running out of memory, please have a look at Predictor.predict_data_mgr_cache_metrics_only() or Predictor.predict_data_mgr() or consider setting cache_preds to False (if not done already)

predict_data_mgr_cache_all(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches all predictions and metrics (yields them in dicts)

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all predictions;
dict – a dictionary containing all validation metrics (maybe empty)

Warning

Since this function caches all predictions and metrics, this may result in huge memory consumption. If you are running out of memory, please have a look at Predictor.predict_data_mgr_cache_metrics_only() or Predictor.predict_data_mgr()

predict_data_mgr_cache_metrics_only(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)¶

Defines a routine to predict data obtained from a batchgenerator and caches the metrics

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all validation metrics (maybe empty)

Notes

This function stores each prediction temporarily for metric calculation; This results in a (typically) way lower memory consumption than Predictor.predict_data_mgr_cache_all(), but still caches the metrics. If this is not desired, it is recommended to use Predictor.predict_data_mgr() and iterate over the generator as this only produces per-batch metrics and predictions and does not cache anything by default

register_callback(callback: delira.training.callbacks.abstract_callback.AbstractCallback)¶

Register Callback to Trainer

Parameters: callback (AbstractCallback) – the callback to register
Raises: AssertionError – callback is not an instance of AbstractCallback and has not both methods [‘at_epoch_begin’, ‘at_epoch_end’]

save_state(file_name, *args, **kwargs)[source]¶

saves the current state via delira.io.tf.save_checkpoint()

Parameters: file_name (str) – filename to save the state to

train(num_epochs, datamgr_train, datamgr_valid=None, val_score_key=None, val_score_mode='highest', reduce_mode='mean', verbose=True)¶

Defines a routine to train a specified number of epochs

Parameters

num_epochs (int) – number of epochs to train
datamgr_train (DataManager) – the datamanager holding the train data
datamgr_valid (DataManager) – the datamanager holding the validation data (default: None)
val_score_key (str) – the key specifying which metric to use for validation (default: None)
val_score_mode (str) – key specifying what kind of validation score is best
reduce_mode (str) – ‘mean’,’sum’,’first_only’
verbose (bool) – whether to show progress bars or not

Raises

NotImplementedError – If not overwritten by subclass

update_state(file_name, *args, **kwargs)¶

Update internal state from a loaded state

Parameters

file_name (str) – file containing the new state to load
*args – positional arguments
**kwargs – keyword arguments

Returns

the trainer with a modified state

Return type

BaseNetworkTrainer

Predictor¶

The predictor implements the basic prediction and metric calculation routines: and can be subclassed for special routines.

It is also the baseclass of all the trainers which extend it’s functionality by training routines

Predictor¶

class Predictor(model, key_mapping: dict, convert_batch_to_npy_fn=<function convert_batch_to_numpy_identity>, prepare_batch_fn=<function Predictor.<lambda>>, **kwargs)[source]¶

Bases: object

Defines an API for Predictions from a Network

See also

PyTorchNetworkTrainer

_Predictor__KEYS_TO_GUARD = []¶

static _Predictor__concatenate_dict_items(dict_like: dict)¶

Function to recursively concatenate dict-items

Parameters: dict_like (dict) – the (nested) dict, whoose items should be concatenated

static _Predictor__convert_dict(old_dict, new_dict)¶

Function to recursively convert dicts

Parameters

old_dict (dict) – the old nested dict
new_dict (dict) – the new nested dict

Returns

the updated new nested dict

Return type

dict

_setup(network, key_mapping, convert_batch_args_kwargs_to_npy_fn, prepare_batch_fn, **kwargs)[source]¶

Parameters

network (AbstractNetwork) – the network to predict from
key_mapping (dict) – a dictionary containing the mapping from the data_dict to the actual model’s inputs. E.g. if a model accepts one input named ‘x’ and the data_dict contains one entry named ‘data’ this argument would have to be {'x': 'data'}
convert_batch_to_npy_fn (type) – a callable function to convert tensors in positional and keyword arguments to numpy
prepare_batch_fn (type) – function converting a batch-tensor to the framework specific tensor-type and pushing it to correct device, default: identity function

static calc_metrics(batch: delira.utils.config.LookupConfig, metrics={}, metric_keys=None)[source]¶

Compute metrics

Parameters

batch (LookupConfig) – dictionary containing the whole batch (including predictions)
metrics (dict) – dict with metrics
metric_keys (dict) – dict of tuples which contains hashables for specifying the items to use for calculating the respective metric. If not specified for a metric, the keys “pred” and “label” are used per default

Returns

dict with metric results

Return type

dict

predict(data: dict, **kwargs)[source]¶

Predict single batch Returns the predictions corresponding to the given data obtained by the model

Parameters

data (dict) – batch dictionary
**kwargs – keyword arguments(directly passed to prepare_batch)

Returns

predicted data

Return type

dict

predict_data_mgr(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)[source]¶

Defines a routine to predict data obtained from a batchgenerator without explicitly caching anything

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all predictions of the current batch
dict – a dictionary containing all metrics of the current batch

predict_data_mgr_cache(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, cache_preds=False, **kwargs)[source]¶

Defines a routine to predict data obtained from a batchgenerator and caches all predictions and metrics (yields them in dicts)

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all validation metrics (maybe empty)
dict – a dictionary containing all predictions; If cache_preds=True

Warning

Since this function caches all metrics and may additionally cache all predictions (based on the argument cache_preds), this may result in huge memory consumption. If you are running out of memory, please have a look at Predictor.predict_data_mgr_cache_metrics_only() or Predictor.predict_data_mgr() or consider setting cache_preds to False (if not done already)

predict_data_mgr_cache_all(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)[source]¶

Defines a routine to predict data obtained from a batchgenerator and caches all predictions and metrics (yields them in dicts)

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all predictions;
dict – a dictionary containing all validation metrics (maybe empty)

Warning

Since this function caches all predictions and metrics, this may result in huge memory consumption. If you are running out of memory, please have a look at Predictor.predict_data_mgr_cache_metrics_only() or Predictor.predict_data_mgr()

predict_data_mgr_cache_metrics_only(datamgr, batchsize=None, metrics={}, metric_keys=None, verbose=False, **kwargs)[source]¶

Defines a routine to predict data obtained from a batchgenerator and caches the metrics

Parameters

datamgr (BaseDataManager) – Manager producing a generator holding the batches
batchsize (int) – Artificial batchsize (sampling will be done with batchsize 1 and sampled data will be stacked to match the artificial batchsize)(default: None)
metrics (dict) – the metrics to calculate
metric_keys (dict) – the batch_dict items to use for metric calculation
verbose (bool) – whether to show a progress-bar or not, default: False
kwargs – keyword arguments passed to prepare_batch_fn()

Yields

dict – a dictionary containing all validation metrics (maybe empty)

Notes

This function stores each prediction temporarily for metric calculation; This results in a (typically) way lower memory consumption than Predictor.predict_data_mgr_cache_all(), but still caches the metrics. If this is not desired, it is recommended to use Predictor.predict_data_mgr() and iterate over the generator as this only produces per-batch metrics and predictions and does not cache anything by default

Experiments¶

Experiments are the outermost class to control your training, it wraps your NetworkTrainer and provides utilities for cross-validation.

BaseExperiment¶

class BaseExperiment(params: Union[str, delira.training.parameters.Parameters], model_cls: delira.models.abstract_network.AbstractNetwork, n_epochs=None, name=None, save_path=None, key_mapping=None, val_score_key=None, optim_builder=None, checkpoint_freq=1, trainer_cls=<class 'delira.training.base_trainer.BaseNetworkTrainer'>, predictor_cls=<class 'delira.training.predictor.Predictor'>, **kwargs)[source]¶

Bases: object

Baseclass for Experiments.

Implements:

Setup-Behavior for Models, Trainers and Predictors (depending on train
and test case)
The K-Fold logic (including stratified and random splitting)
Argument Handling

_resolve_kwargs(kwargs: Optional[dict])[source]¶

Merges given kwargs with self.kwargs If same argument is present in both kwargs, the one from the given kwargs will be used here

Parameters: kwargs (dict) – the given kwargs to merge with self.kwargs
Returns: merged kwargs
Return type: dict

_resolve_params(params: Optional[delira.training.parameters.Parameters])[source]¶

Merges the given params with self.params. If the same argument is given in both params, the one from the currently given parameters is used here

Parameters: params (Parameters or None) – the parameters to merge with self.params
Returns: the merged parameter instance
Return type: Parameters

_setup_test(params, model, convert_batch_to_npy_fn, prepare_batch_fn, **kwargs)[source]¶

Parameters

params (Parameters) – the parameters containing the model and training kwargs (ignored here, just passed for subclassing and unified API)
model (AbstractNetwork) – the model to test
convert_batch_to_npy_fn (function) – function to convert a batch of tensors to numpy
prepare_batch_fn (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices
**kwargs – additional keyword arguments

Returns

the created predictor

Return type

Predictor

_setup_training(params, **kwargs)[source]¶

Handles the setup for training case

Parameters

params (Parameters) – the parameters containing the model and training kwargs
**kwargs – additional keyword arguments

Returns

the created trainer

Return type

BaseNetworkTrainer

kfold(data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, num_epochs=None, num_splits=None, shuffle=False, random_seed=None, split_type='random', val_split=0.2, label_key='label', train_kwargs: dict = None, metric_keys: dict = None, test_kwargs: dict = None, params=None, verbose=False, **kwargs)[source]¶

Performs a k-Fold cross-validation

Parameters

data (BaseDataManager) – the data to use for training(, validation) and testing. Will be split based on split_type and val_split
metrics (dict) – dictionary containing the metrics to evaluate during k-fold
num_epochs (int or None) – number of epochs to train (if not given, will either be extracted from params, self.parms or self.n_epochs)
num_splits (int or None) – the number of splits to extract from data. If None: uses a default of 10
shuffle (bool) – whether to shuffle the data before splitting or not (implemented by index-shuffling rather than actual data-shuffling to retain potentially lazy-behavior of datasets)
random_seed (None) – seed to seed numpy, the splitting functions and the used backend-framework
split_type (str) – must be one of [‘random’, ‘stratified’] if ‘random’: uses random data splitting if ‘stratified’: uses stratified data splitting. Stratification will be based on label_key
val_split (float or None) – the fraction of the train data to use as validation set. If None: No validation will be done during training; only testing for each fold after the training is complete
label_key (str) – the label to use for stratification. Will be ignored unless split_type is ‘stratified’. Default: ‘label’
train_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the train data. If None: empty dict will be passed
metric_keys (dict of tuples) – the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label will be used for metric calculation
test_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the test and validation data. If None: empty dict will be passed
params (:class:`Parameters`or None) – the training and model parameters (will be merged with self.params)
verbose (bool) – verbosity
**kwargs – additional keyword arguments

Returns

dict – all predictions from all folds
dict – all metric values from all folds

Raises

ValueError – if split_type is neither ‘random’, nor ‘stratified’

See also

sklearn.model_selection.KFold

and sklearn.model_selection.ShuffleSplit for random data-splitting

sklearn.model_selection.StratifiedKFold

and sklearn.model_selection.StratifiedShuffleSplit for stratified data-splitting

BaseDataManager.update_from_state_dict() for updating the

data managers by kwargs

BaseExperiment.run() for the training
BaseExperiment.test() for the testing

Notes

using stratified splits may be slow during split-calculation, since each item must be loaded once to obtain the labels necessary for stratification.

static load(file_name)[source]¶

Loads whole experiment

Parameters: file_name (str) – file_name to load the experiment from

resume(save_path: str, train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)[source]¶

Resumes a previous training by passing an explicit save_path instead of generating a new one

Parameters

save_path (str) – path to previous training
train_data (BaseDataManager) – the data to use for training
val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None
params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)
**kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

run(train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)[source]¶

Setup and run training

Parameters

train_data (BaseDataManager) – the data to use for training
val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None
params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)
**kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

save()[source]¶: Saves the Whole experiments

setup(params, training=True, **kwargs)[source]¶

Defines the setup behavior (model, trainer etc.) for training and testing case

Parameters

params (Parameters) – the parameters to use for setup
training (bool) – whether to setup for training case or for testing case
**kwargs – additional keyword arguments

Returns

BaseNetworkTrainer – the created trainer (if training=True)
Predictor – the created predictor (if training=False)

See also

BaseExperiment._setup_training() for training setup
BaseExperiment._setup_test() for test setup

test(network, test_data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, metric_keys=None, verbose=False, prepare_batch=<function BaseExperiment.<lambda>>, convert_fn=<function BaseExperiment.<lambda>>, **kwargs)[source]¶

Setup and run testing on a given network

Parameters

network (AbstractNetwork) – the (trained) network to test
test_data (BaseDataManager) – the data to use for testing
metrics (dict) – the metrics to calculate
metric_keys (dict of tuples) – the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label will be used for metric calculation
verbose (bool) – verbosity of the test process
prepare_batch (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices
convert_fn (function) – function to convert a batch of tensors to numpy
**kwargs – additional keyword arguments

Returns

dict – all predictions obtained by feeding the test_data through the network
dict – all metrics calculated upon the test_data and the obtained predictions

PyTorchExperiment¶

class PyTorchExperiment(params: Union[str, delira.training.parameters.Parameters], model_cls: delira.models.abstract_network.AbstractPyTorchNetwork, n_epochs=None, name=None, save_path=None, key_mapping=None, val_score_key=None, optim_builder=<function create_optims_default_pytorch>, checkpoint_freq=1, trainer_cls=<class 'delira.training.pytorch_trainer.PyTorchNetworkTrainer'>, **kwargs)[source]¶

Bases: delira.training.experiment.BaseExperiment

_resolve_kwargs(kwargs: Optional[dict])¶

Merges given kwargs with self.kwargs If same argument is present in both kwargs, the one from the given kwargs will be used here

Parameters: kwargs (dict) – the given kwargs to merge with self.kwargs
Returns: merged kwargs
Return type: dict

_resolve_params(params: Optional[delira.training.parameters.Parameters])¶

Merges the given params with self.params. If the same argument is given in both params, the one from the currently given parameters is used here

Parameters: params (Parameters or None) – the parameters to merge with self.params
Returns: the merged parameter instance
Return type: Parameters

_setup_test(params, model, convert_batch_to_npy_fn, prepare_batch_fn, **kwargs)¶

Parameters

params (Parameters) – the parameters containing the model and training kwargs (ignored here, just passed for subclassing and unified API)
model (AbstractNetwork) – the model to test
convert_batch_to_npy_fn (function) – function to convert a batch of tensors to numpy
prepare_batch_fn (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices
**kwargs – additional keyword arguments

Returns

the created predictor

Return type

Predictor

_setup_training(params, **kwargs)¶

Handles the setup for training case

Parameters

params (Parameters) – the parameters containing the model and training kwargs
**kwargs – additional keyword arguments

Returns

the created trainer

Return type

BaseNetworkTrainer

kfold(data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, num_epochs=None, num_splits=None, shuffle=False, random_seed=None, split_type='random', val_split=0.2, label_key='label', train_kwargs: dict = None, test_kwargs: dict = None, metric_keys: dict = None, params=None, verbose=False, **kwargs)[source]¶

Performs a k-Fold cross-validation

Parameters

data (BaseDataManager) – the data to use for training(, validation) and testing. Will be split based on split_type and val_split
metrics (dict) – dictionary containing the metrics to evaluate during k-fold
num_epochs (int or None) – number of epochs to train (if not given, will either be extracted from params, self.parms or self.n_epochs)
num_splits (int or None) – the number of splits to extract from data. If None: uses a default of 10
shuffle (bool) – whether to shuffle the data before splitting or not (implemented by index-shuffling rather than actual data-shuffling to retain potentially lazy-behavior of datasets)
random_seed (None) – seed to seed numpy, the splitting functions and the used backend-framework
split_type (str) – must be one of [‘random’, ‘stratified’] if ‘random’: uses random data splitting if ‘stratified’: uses stratified data splitting. Stratification will be based on label_key
val_split (float or None) – the fraction of the train data to use as validation set. If None: No validation will be done during training; only testing for each fold after the training is complete
label_key (str) – the label to use for stratification. Will be ignored unless split_type is ‘stratified’. Default: ‘label’
train_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the train data. If None: empty dict will be passed
metric_keys (dict of tuples) – the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label will be used for metric calculation
test_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the test and validation data. If None: empty dict will be passed
params (:class:`Parameters`or None) – the training and model parameters (will be merged with self.params)
verbose (bool) – verbosity
**kwargs – additional keyword arguments

Returns

dict – all predictions from all folds
dict – all metric values from all folds

Raises

ValueError – if split_type is neither ‘random’, nor ‘stratified’

See also

sklearn.model_selection.KFold

and sklearn.model_selection.ShuffleSplit for random data-splitting

sklearn.model_selection.StratifiedKFold

and sklearn.model_selection.StratifiedShuffleSplit for stratified data-splitting

BaseDataManager.update_from_state_dict() for updating the

data managers by kwargs

BaseExperiment.run() for the training
BaseExperiment.test() for the testing

Notes

using stratified splits may be slow during split-calculation, since each item must be loaded once to obtain the labels necessary for stratification.

static load(file_name)¶

Loads whole experiment

Parameters: file_name (str) – file_name to load the experiment from

resume(save_path: str, train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)¶

Resumes a previous training by passing an explicit save_path instead of generating a new one

Parameters

save_path (str) – path to previous training
train_data (BaseDataManager) – the data to use for training
val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None
params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)
**kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

run(train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)¶

Setup and run training

Parameters

train_data (BaseDataManager) – the data to use for training
val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None
params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)
**kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

save()¶: Saves the Whole experiments

setup(params, training=True, **kwargs)¶

Defines the setup behavior (model, trainer etc.) for training and testing case

Parameters

params (Parameters) – the parameters to use for setup
training (bool) – whether to setup for training case or for testing case
**kwargs – additional keyword arguments

Returns

BaseNetworkTrainer – the created trainer (if training=True)
Predictor – the created predictor (if training=False)

See also

BaseExperiment._setup_training() for training setup
BaseExperiment._setup_test() for test setup

test(network, test_data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, metric_keys=None, verbose=False, prepare_batch=None, convert_fn=None, **kwargs)[source]¶

Setup and run testing on a given network

Parameters

network (AbstractNetwork) – the (trained) network to test
test_data (BaseDataManager) – the data to use for testing
metrics (dict) – the metrics to calculate
metric_keys (dict of tuples) –
the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label

will be used for metric calculation
verbose (bool) – verbosity of the test process
prepare_batch (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices. If not further specified uses the network’s prepare_batch with CPU devices
convert_fn (function) – function to convert a batch of tensors to numpy if not specified defaults to convert_torch_tensor_to_npy()
**kwargs – additional keyword arguments

Returns

dict – all predictions obtained by feeding the test_data through the network
dict – all metrics calculated upon the test_data and the obtained predictions

TfExperiment¶

class TfExperiment(params: Union[str, delira.training.parameters.Parameters], model_cls: delira.models.abstract_network.AbstractTfNetwork, n_epochs=None, name=None, save_path=None, key_mapping=None, val_score_key=None, optim_builder=<function create_optims_default_tf>, checkpoint_freq=1, trainer_cls=<class 'delira.training.tf_trainer.TfNetworkTrainer'>, **kwargs)[source]¶

Bases: delira.training.experiment.BaseExperiment

_resolve_kwargs(kwargs: Optional[dict])¶

Merges given kwargs with self.kwargs If same argument is present in both kwargs, the one from the given kwargs will be used here

Parameters: kwargs (dict) – the given kwargs to merge with self.kwargs
Returns: merged kwargs
Return type: dict

_resolve_params(params: Optional[delira.training.parameters.Parameters])¶

Merges the given params with self.params. If the same argument is given in both params, the one from the currently given parameters is used here

Parameters: params (Parameters or None) – the parameters to merge with self.params
Returns: the merged parameter instance
Return type: Parameters

_setup_test(params, model, convert_batch_to_npy_fn, prepare_batch_fn, **kwargs)¶

Parameters

params (Parameters) – the parameters containing the model and training kwargs (ignored here, just passed for subclassing and unified API)
model (AbstractNetwork) – the model to test
convert_batch_to_npy_fn (function) – function to convert a batch of tensors to numpy
prepare_batch_fn (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices
**kwargs – additional keyword arguments

Returns

the created predictor

Return type

Predictor

_setup_training(params, **kwargs)¶

Handles the setup for training case

Parameters

params (Parameters) – the parameters containing the model and training kwargs
**kwargs – additional keyword arguments

Returns

the created trainer

Return type

BaseNetworkTrainer

kfold(data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, num_epochs=None, num_splits=None, shuffle=False, random_seed=None, split_type='random', val_split=0.2, label_key='label', train_kwargs: dict = None, test_kwargs: dict = None, metric_keys: dict = None, params=None, verbose=False, **kwargs)[source]¶

Performs a k-Fold cross-validation

Parameters

data (BaseDataManager) – the data to use for training(, validation) and testing. Will be split based on split_type and val_split
metrics (dict) – dictionary containing the metrics to evaluate during k-fold
num_epochs (int or None) – number of epochs to train (if not given, will either be extracted from params, self.parms or self.n_epochs)
num_splits (int or None) – the number of splits to extract from data. If None: uses a default of 10
shuffle (bool) – whether to shuffle the data before splitting or not (implemented by index-shuffling rather than actual data-shuffling to retain potentially lazy-behavior of datasets)
random_seed (None) – seed to seed numpy, the splitting functions and the used backend-framework
split_type (str) – must be one of [‘random’, ‘stratified’] if ‘random’: uses random data splitting if ‘stratified’: uses stratified data splitting. Stratification will be based on label_key
val_split (float or None) – the fraction of the train data to use as validation set. If None: No validation will be done during training; only testing for each fold after the training is complete
label_key (str) – the label to use for stratification. Will be ignored unless split_type is ‘stratified’. Default: ‘label’
train_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the train data. If None: empty dict will be passed
metric_keys (dict of tuples) – the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label will be used for metric calculation
test_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the test and validation data. If None: empty dict will be passed
params (:class:`Parameters`or None) – the training and model parameters (will be merged with self.params)
verbose (bool) – verbosity
**kwargs – additional keyword arguments

Returns

dict – all predictions from all folds
dict – all metric values from all folds

Raises

ValueError – if split_type is neither ‘random’, nor ‘stratified’

See also

sklearn.model_selection.KFold

and sklearn.model_selection.ShuffleSplit for random data-splitting

sklearn.model_selection.StratifiedKFold

and sklearn.model_selection.StratifiedShuffleSplit for stratified data-splitting

BaseDataManager.update_from_state_dict() for updating the

data managers by kwargs

BaseExperiment.run() for the training
BaseExperiment.test() for the testing

Notes

using stratified splits may be slow during split-calculation, since each item must be loaded once to obtain the labels necessary for stratification.

static load(file_name)¶

Loads whole experiment

Parameters: file_name (str) – file_name to load the experiment from

resume(save_path: str, train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)¶

Resumes a previous training by passing an explicit save_path instead of generating a new one

Parameters

save_path (str) – path to previous training
train_data (BaseDataManager) – the data to use for training
val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None
params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)
**kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

run(train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)¶

Setup and run training

Parameters

train_data (BaseDataManager) – the data to use for training
val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None
params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)
**kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

save()¶: Saves the Whole experiments

setup(params, training=True, **kwargs)[source]¶

Defines the setup behavior (model, trainer etc.) for training and testing case

Parameters

params (Parameters) – the parameters to use for setup
training (bool) – whether to setup for training case or for testing case
**kwargs – additional keyword arguments

Returns

BaseNetworkTrainer – the created trainer (if training=True)
Predictor – the created predictor (if training=False)

See also

BaseExperiment._setup_training() for training setup
BaseExperiment._setup_test() for test setup

test(network, test_data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, metric_keys=None, verbose=False, prepare_batch=<function TfExperiment.<lambda>>, convert_fn=None, **kwargs)[source]¶

Setup and run testing on a given network

Parameters

network (AbstractNetwork) – the (trained) network to test
test_data (BaseDataManager) – the data to use for testing
metrics (dict) – the metrics to calculate
metric_keys (dict of tuples) –
the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label

will be used for metric calculation
verbose (bool) – verbosity of the test process
prepare_batch (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices. If not further specified uses the network’s prepare_batch with CPU devices
convert_fn (function) – function to convert a batch of tensors to numpy if not specified defaults to convert_torch_tensor_to_npy()
**kwargs – additional keyword arguments

Returns

dict – all predictions obtained by feeding the test_data through the network
dict – all metrics calculated upon the test_data and the obtained predictions

Callbacks¶

Callbacks are essential to provide a uniform API for tasks like early stopping etc. The PyTorch learning rate schedulers are also implemented as callbacks. Every callback should ber derived from AbstractCallback and must provide the methods at_epoch_begin and at_epoch_end.

AbstractCallback¶

class AbstractCallback(*args, **kwargs)[source]¶

Bases: object

Implements abstract callback interface. All callbacks should be derived from this class

See also

AbstractNetworkTrainer

at_epoch_begin(trainer, **kwargs)[source]¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)[source]¶

Function which will be executed at end of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

EarlyStopping¶

class EarlyStopping(monitor_key, min_delta=0, patience=0, mode='min')[source]¶

Bases: delira.training.callbacks.abstract_callback.AbstractCallback

Implements Early Stopping as callback

See also

AbstractCallback

_is_better(metric)[source]¶

Helper function to decide whether the current metric is better than the best metric so far

Parameters: metric – current metric value
Returns: whether this metric is the new best metric or not
Return type: bool

at_epoch_begin(trainer, **kwargs)¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)[source]¶

Actual early stopping: Checks at end of each epoch if monitored metric is new best and if it hasn’t improved over self.patience epochs, the training will be stopped

Parameters

trainer (AbstractNetworkTrainer) – the trainer whose arguments can be modified
**kwargs – additional keyword arguments

Returns

trainer with modified attributes

Return type

AbstractNetworkTrainer

DefaultPyTorchSchedulerCallback¶

class DefaultPyTorchSchedulerCallback(*args, **kwargs)[source]¶

Bases: delira.training.callbacks.abstract_callback.AbstractCallback

Implements a Callback, which at_epoch_end function is suitable for most schedulers

at_epoch_begin(trainer, **kwargs)¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)[source]¶

Executes a single scheduling step

Parameters

trainer (PyTorchNetworkTrainer) – the trainer class, which can be changed
**kwargs – additional keyword arguments

Returns

modified trainer

Return type

PyTorchNetworkTrainer

CosineAnnealingLRCallback¶

class CosineAnnealingLRCallback(optimizer, T_max, eta_min=0, last_epoch=-1)[source]¶

Bases: delira.training.callbacks.pytorch_schedulers.DefaultPyTorchSchedulerCallback

Wraps PyTorch’s CosineAnnealingLR Scheduler as callback

at_epoch_begin(trainer, **kwargs)¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)¶

Executes a single scheduling step

Parameters

trainer (PyTorchNetworkTrainer) – the trainer class, which can be changed
**kwargs – additional keyword arguments

Returns

modified trainer

Return type

PyTorchNetworkTrainer

ExponentialLRCallback¶

class ExponentialLRCallback(optimizer, gamma, last_epoch=-1)[source]¶

Bases: delira.training.callbacks.pytorch_schedulers.DefaultPyTorchSchedulerCallback

Wraps PyTorch’s ExponentialLR Scheduler as callback

at_epoch_begin(trainer, **kwargs)¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)¶

Executes a single scheduling step

Parameters

trainer (PyTorchNetworkTrainer) – the trainer class, which can be changed
**kwargs – additional keyword arguments

Returns

modified trainer

Return type

PyTorchNetworkTrainer

LambdaLRCallback¶

class LambdaLRCallback(optimizer, lr_lambda, last_epoch=-1)[source]¶

Bases: delira.training.callbacks.pytorch_schedulers.DefaultPyTorchSchedulerCallback

Wraps PyTorch’s LambdaLR Scheduler as callback

at_epoch_begin(trainer, **kwargs)¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)¶

Executes a single scheduling step

Parameters

trainer (PyTorchNetworkTrainer) – the trainer class, which can be changed
**kwargs – additional keyword arguments

Returns

modified trainer

Return type

PyTorchNetworkTrainer

MultiStepLRCallback¶

class MultiStepLRCallback(optimizer, milestones, gamma=0.1, last_epoch=-1)[source]¶

Bases: delira.training.callbacks.pytorch_schedulers.DefaultPyTorchSchedulerCallback

Wraps PyTorch’s MultiStepLR Scheduler as callback

at_epoch_begin(trainer, **kwargs)¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)¶

Executes a single scheduling step

Parameters

trainer (PyTorchNetworkTrainer) – the trainer class, which can be changed
**kwargs – additional keyword arguments

Returns

modified trainer

Return type

PyTorchNetworkTrainer

ReduceLROnPlateauCallback¶

class ReduceLROnPlateauCallback(optimizer, mode='min', factor=0.1, patience=10, verbose=False, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)[source]¶

Bases: delira.training.callbacks.pytorch_schedulers.DefaultPyTorchSchedulerCallback

Wraps PyTorch’s ReduceLROnPlateau Scheduler as Callback

at_epoch_begin(trainer, **kwargs)¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)[source]¶

Executes a single scheduling step

Parameters

trainer (PyTorchNetworkTrainer) – the trainer class, which can be changed
kwargs – additional keyword arguments

Returns

modified trainer

Return type

PyTorchNetworkTrainer

StepLRCallback¶

class StepLRCallback(optimizer, step_size, gamma=0.1, last_epoch=-1)[source]¶

Bases: delira.training.callbacks.pytorch_schedulers.DefaultPyTorchSchedulerCallback

Wraps PyTorch’s StepLR Scheduler as callback

at_epoch_begin(trainer, **kwargs)¶

Function which will be executed at begin of each epoch

Parameters

trainer (AbstractNetworkTrainer) –
**kwargs – additional keyword arguments

Returns

modified trainer attributes, where the name must correspond to the trainer’s attribute name

Return type

dict

at_epoch_end(trainer, **kwargs)¶

Executes a single scheduling step

Parameters

trainer (PyTorchNetworkTrainer) – the trainer class, which can be changed
**kwargs – additional keyword arguments

Returns

modified trainer

Return type

PyTorchNetworkTrainer

Custom Loss Functions¶

BCEFocalLossPyTorch¶

class BCEFocalLossPyTorch(alpha=None, gamma=2, reduction='elementwise_mean')[source]¶

Bases: torch.nn.Module

Focal loss for binary case without(!) logit

forward(p, t)[source]¶

BCEFocalLossLogitPyTorch¶

class BCEFocalLossLogitPyTorch(alpha=None, gamma=2, reduction='elementwise_mean')[source]¶

Bases: torch.nn.Module

Focal loss for binary case WITH logit

forward(p, t)[source]¶

Metrics¶

SklearnClassificationMetric¶

class SklearnClassificationMetric(score_fn, gt_logits=False, pred_logits=True, **kwargs)[source]¶: Bases: object

SklearnAccuracyScore¶

class SklearnAccuracyScore(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Accuracy Metric

SklearnBalancedAccuracyScore¶

class SklearnBalancedAccuracyScore(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Balanced Accuracy Metric

SklearnF1Score¶

class SklearnF1Score(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

F1 Score

SklearnFBetaScore¶

class SklearnFBetaScore(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

F-Beta Score (Generalized F1)

SklearnHammingLoss¶

class SklearnHammingLoss(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Hamming Loss

SklearnJaccardSimilarityScore¶

class SklearnJaccardSimilarityScore(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Jaccard Similarity Score

SklearnLogLoss¶

class SklearnLogLoss(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Log Loss (NLL)

SklearnMatthewsCorrCoeff¶

class SklearnMatthewsCorrCoeff(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Matthews Correlation Coefficient

SklearnPrecisionScore¶

class SklearnPrecisionScore(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Precision Score

SklearnRecallScore¶

class SklearnRecallScore(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Recall Score

SklearnZeroOneLoss¶

class SklearnZeroOneLoss(gt_logits=False, pred_logits=True, **kwargs)[source]¶

Bases: delira.training.metrics.SklearnClassificationMetric

Zero One Loss

AurocMetric¶

class AurocMetric(classes=(0, 1), **kwargs)[source]¶: Bases: object

convert_batch_to_numpy_identity¶

convert_batch_to_numpy_identity(*args, **kwargs)[source]¶

Corrects the shape of all zero-sized numpy arrays to be at least 1d

Parameters

*args – positional arguments of potential arrays to be corrected
**kwargs – keyword arguments of potential arrays to be corrected

float_to_pytorch_tensor¶

float_to_pytorch_tensor(f: float)[source]¶

Converts a single float to a PyTorch Float-Tensor

Parameters: f (float) – float to convert
Returns: converted float
Return type: torch.Tensor

create_optims_default_pytorch¶

create_optims_default_pytorch(model, optim_cls, **optim_params)[source]¶

Function to create a optimizer dictionary (in this case only one optimizer for the whole network)

Parameters

model (AbstractPyTorchNetwork) – model whose parameters should be updated by the optimizer
optim_cls – Class implementing an optimization algorithm
**optim_params – Additional keyword arguments (passed to the optimizer class

Returns

dictionary containing all created optimizers

Return type

dict

create_optims_gan_pytorch¶

convert_torch_tensor_to_npy¶

convert_torch_tensor_to_npy(*args, **kwargs)[source]¶

Function to convert all torch Tensors to numpy arrays and reshape zero-size tensors

Parameters

*args – arbitrary positional arguments
**kwargs – arbitrary keyword arguments

Returns

Iterable – all given positional arguments (converted if necessary)
dict – all given keyword arguments (converted if necessary)

create_optims_default_tf¶

create_optims_default_tf(optim_cls, **optim_params)[source]¶

Function to create a optimizer dictionary (in this case only one optimizer)

Parameters

optim_cls – Class implementing an optimization algorithm
**optim_params – Additional keyword arguments (passed to the optimizer class)

Returns

dictionary containing all created optimizers

Return type

dict

initialize_uninitialized¶

initialize_uninitialized(sess)[source]¶

Function to initialize only uninitialized variables in a session graph

Parameters: sess (tf.Session()) –

convert_tf_tensor_to_npy¶

convert_tf_tensor_to_npy(*args, **kwargs)[source]¶

Function to convert all tf Tensors to numpy arrays and reshape zero-size tensors

Parameters

*args – arbitrary positional arguments
**kwargs – arbitrary keyword arguments

Returns

Iterable – all given positional arguments (converted if necessary)
dict – all given keyword arguments (converted if necessary)

Utils¶

This package provides utility functions as image operations, various decorators, path operations and time operations.

class DebugDisabled[source]¶

Bases: delira.utils.context_managers.DebugMode

Context Manager to disable the debug mode for the wrapped context

_switch_to_new_mode()¶: helper function to switch to the new debug mode (and saving the previous one in self._mode)

class DebugEnabled[source]¶

Bases: delira.utils.context_managers.DebugMode

Context Manager to enable the debug mode for the wrapped context

_switch_to_new_mode()¶: helper function to switch to the new debug mode (and saving the previous one in self._mode)

class DebugMode(mode)[source]¶

Bases: object

Context Manager to set a specific debug mode for the code inside the defined context (and reverting to previous mode afterwards)

_switch_to_new_mode()[source]¶: helper function to switch to the new debug mode (and saving the previous one in self._mode)

class DefaultOptimWrapperTorch(optimizer: torch.optim.Optimizer, *args, **kwargs)[source]¶

Bases: object

Class wrapping a torch optimizer to mirror the behavior of apex without depending on it

add_param_group(param_group)[source]¶

load_state_dict(state_dict)[source]¶

scale_loss(loss)[source]¶

Function which scales the loss in apex and yields the unscaled loss here to mirror the API

Parameters: loss (torch.Tensor) – the unscaled loss

state_dict()[source]¶

step(closure=None)[source]¶

Wraps the step method of the optimizer and calls the original step method

Parameters: closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.

zero_grad()[source]¶

classtype_func(class_object)[source]¶

Decorator to Check whether the first argument of the decorated function is a subclass of a certain type

Parameters: class_object (Any) – type the first function argument should be subclassed from
Returns
Return type: Wrapped Function
Raises: AssertionError – First argument of decorated function is not a subclass of given type

dtype_func(class_object)[source]¶

Decorator to Check whether the first argument of the decorated function is of a certain type

Parameters: class_object (Any) – type the first function argument should have
Returns
Return type: Wrapped Function
Raises: AssertionError – First argument of decorated function is not of given type

make_deprecated(new_func)[source]¶

Decorator which raises a DeprecationWarning for the decorated object

Parameters: new_func (Any) – new function which should be used instead of the decorated one
Returns
Return type: Wrapped Function
Raises: Deprecation Warning –

numpy_array_func(func)¶

torch_module_func(func)¶

torch_tensor_func(func)¶

bounding_box(mask, margin=None)[source]¶

Calculate bounding box coordinates of binary mask

Parameters

mask (SimpleITK.Image) – Binary mask
margin (int, default: None) – margin to be added to min/max on each dimension

Returns

bounding box coordinates of the form (xmin, xmax, ymin, ymax, zmin, zmax)

Return type

tuple

calculate_origin_offset(new_spacing, old_spacing)[source]¶

Calculates the origin offset of two spacings

Parameters

new_spacing (list or np.ndarray or tuple) – new spacing
old_spacing (list or np.ndarray or tuple) – old spacing

Returns

origin offset

Return type

np.ndarray

max_energy_slice(img)[source]¶

Determine the axial slice in which the image energy is max

Parameters: img (SimpleITK.Image) – given image
Returns: slice index
Return type: int

sitk_copy_metadata(img_source, img_target)[source]¶

Copy metadata (=DICOM Tags) from one image to another

Parameters

img_source (SimpleITK.Image) – Source image
img_target (SimpleITK.Image) – Source image

Returns

Target image with copied metadata

Return type

SimpleITK.Image

sitk_img_func(func)¶

sitk_new_blank_image(size, spacing, direction, origin, default_value=0.0)[source]¶

Create a new blank image with given properties

Parameters

size (list or np.ndarray or tuple) – new image size
spacing (list or np.ndarray or tuple) – spacing of new image
direction – new image’s direction
origin – new image’s origin
default_value (float) – new image’s default value

Returns

Blank image with given properties

Return type

SimpleITK.Image

sitk_resample_to_image(image, reference_image, default_value=0.0, interpolator=SimpleITK.sitkLinear, transform=None, output_pixel_type=None)[source]¶

Resamples Image to reference image

Parameters

image (SimpleITK.Image) – the image which should be resampled
reference_image (SimpleITK.Image) – the resampling target
default_value (float) – default value
interpolator (Any) – implements the actual interpolation
transform (Any (default: None)) – transformation
output_pixel_type (Any (default:None)) – type of output pixels

Returns

resampled image

Return type

SimpleITK.Image

sitk_resample_to_shape(img, x, y, z, order=1)[source]¶

Resamples Image to given shape

Parameters

img (SimpleITK.Image) –
x (int) – shape in x-direction
y (int) – shape in y-direction
z (int) – shape in z-direction
order (int) – interpolation order

Returns

Resampled Image

Return type

SimpleITK.Image

sitk_resample_to_spacing(image, new_spacing=(1.0, 1.0, 1.0), interpolator=SimpleITK.sitkLinear, default_value=0.0)[source]¶

Resamples SITK Image to a given spacing

Parameters

image (SimpleITK.Image) – image which should be resampled
new_spacing (list or np.ndarray or tuple) – target spacing
interpolator (Any) – implements the actual interpolation
default_value (float) – default value

Returns

resampled Image with target spacing

Return type

SimpleITK.Image

subdirs(d)[source]¶

For a given directory, return a list of all subdirectories (full paths)

Parameters: d (string) – given root directory
Returns: list of strings of all subdirectories
Return type: list

now()[source]¶

Return current time as YYYY-MM-DD_HH-MM-SS

Returns: current time
Return type: string

class LookupConfig(*args, **kwargs)[source]¶

Bases: trixi.util.Config

Helper class to have nested lookups in all subdicts of Config

nested_get(key, *args, **kwargs)[source]¶

Returns all occurances of key in self and subdicts

Parameters

key (str) – the key to search for
*args – positional arguments to provide default value
**kwargs – keyword arguments to provide default value

Raises

KeyError – Multiple Values are found for key (unclear which value should be returned) OR No Value was found for key and no default value was given

Returns

value corresponding to key (or default if value was not found)

Return type

Any

Backend Resolution¶

These functions are used to determine the installed backends and update the created config file. They also need to be used, to guard backend specific code,