Experiments

Experiments are the outermost class to control your training, it wraps your NetworkTrainer and provides utilities for cross-validation.

BaseExperiment

class BaseExperiment(params: Union[str, delira.training.parameters.Parameters], model_cls: delira.models.abstract_network.AbstractNetwork, n_epochs=None, name=None, save_path=None, key_mapping=None, val_score_key=None, optim_builder=None, checkpoint_freq=1, trainer_cls=<class 'delira.training.base_trainer.BaseNetworkTrainer'>, predictor_cls=<class 'delira.training.predictor.Predictor'>, **kwargs)

Bases: object

Baseclass for Experiments.

Implements:

• Setup-Behavior for Models, Trainers and Predictors (depending on train

  and test case)

• The K-Fold logic (including stratified and random splitting)

• Argument Handling

_resolve_kwargs(kwargs: Optional[dict])

Merges given kwargs with self.kwargs If same argument is present in both kwargs, the one from the given kwargs will be used here

Parameters

kwargs (dict) – the given kwargs to merge with self.kwargs

Returns

merged kwargs

Return type

dict

_resolve_params(params: Optional[delira.training.parameters.Parameters])

Merges the given params with self.params. If the same argument is given in both params, the one from the currently given parameters is used here

Parameters

params (Parameters or None) – the parameters to merge with self.params

Returns

the merged parameter instance

Return type

Parameters

_setup_test(params, model, convert_batch_to_npy_fn, prepare_batch_fn, **kwargs)

Parameters

• params (Parameters) – the parameters containing the model and training kwargs (ignored here, just passed for subclassing and unified API)

• model (AbstractNetwork) – the model to test

• convert_batch_to_npy_fn (function) – function to convert a batch of tensors to numpy

• prepare_batch_fn (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices

• **kwargs – additional keyword arguments

Returns

the created predictor

Return type

Predictor

_setup_training(params, **kwargs)

Handles the setup for training case

Parameters

• params (Parameters) – the parameters containing the model and training kwargs

• **kwargs – additional keyword arguments

Returns

the created trainer

Return type

BaseNetworkTrainer

kfold(data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, num_epochs=None, num_splits=None, shuffle=False, random_seed=None, split_type='random', val_split=0.2, label_key='label', train_kwargs: dict = None, metric_keys: dict = None, test_kwargs: dict = None, params=None, verbose=False, **kwargs)

Performs a k-Fold cross-validation

Parameters

• data (BaseDataManager) – the data to use for training(, validation) and testing. Will be split based on split_type and val_split

• metrics (dict) – dictionary containing the metrics to evaluate during k-fold

• num_epochs (int or None) – number of epochs to train (if not given, will either be extracted from params, self.parms or self.n_epochs)

• num_splits (int or None) – the number of splits to extract from data. If None: uses a default of 10

• shuffle (bool) – whether to shuffle the data before splitting or not (implemented by index-shuffling rather than actual data-shuffling to retain potentially lazy-behavior of datasets)

• random_seed (None) – seed to seed numpy, the splitting functions and the used backend-framework

• split_type (str) – must be one of [‘random’, ‘stratified’] if ‘random’: uses random data splitting if ‘stratified’: uses stratified data splitting. Stratification will be based on label_key

• val_split (float or None) – the fraction of the train data to use as validation set. If None: No validation will be done during training; only testing for each fold after the training is complete

• label_key (str) – the label to use for stratification. Will be ignored unless split_type is ‘stratified’. Default: ‘label’

• train_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the train data. If None: empty dict will be passed

• metric_keys (dict of tuples) – the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label will be used for metric calculation

• test_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the test and validation data. If None: empty dict will be passed

• params (:class:`Parameters`or None) – the training and model parameters (will be merged with self.params)

• verbose (bool) – verbosity

• **kwargs – additional keyword arguments

Returns

• dict – all predictions from all folds

• dict – all metric values from all folds

Raises

ValueError – if split_type is neither ‘random’, nor ‘stratified’

See also

• sklearn.model_selection.KFold

and sklearn.model_selection.ShuffleSplit for random data-splitting

• sklearn.model_selection.StratifiedKFold

and sklearn.model_selection.StratifiedShuffleSplit for stratified data-splitting

• BaseDataManager.update_from_state_dict() for updating the

data managers by kwargs

• BaseExperiment.run() for the training

• BaseExperiment.test() for the testing

Notes

using stratified splits may be slow during split-calculation, since each item must be loaded once to obtain the labels necessary for stratification.

static load(file_name)

Loads whole experiment

Parameters

file_name (str) – file_name to load the experiment from

resume(save_path: str, train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)

Resumes a previous training by passing an explicit save_path instead of generating a new one

Parameters

• save_path (str) – path to previous training

• train_data (BaseDataManager) – the data to use for training

• val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None

• params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)

• **kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

run(train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)

Setup and run training

Parameters

• train_data (BaseDataManager) – the data to use for training

• val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None

• params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)

• **kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

save()

Saves the Whole experiments

setup(params, training=True, **kwargs)

Defines the setup behavior (model, trainer etc.) for training and testing case

Parameters

• params (Parameters) – the parameters to use for setup

• training (bool) – whether to setup for training case or for testing case

• **kwargs – additional keyword arguments

Returns

• BaseNetworkTrainer – the created trainer (if training=True)

• Predictor – the created predictor (if training=False)

See also

• BaseExperiment._setup_training() for training setup

• BaseExperiment._setup_test() for test setup

test(network, test_data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, metric_keys=None, verbose=False, prepare_batch=<function BaseExperiment.<lambda>>, convert_fn=<function BaseExperiment.<lambda>>, **kwargs)

Setup and run testing on a given network

Parameters

• network (AbstractNetwork) – the (trained) network to test

• test_data (BaseDataManager) – the data to use for testing

• metrics (dict) – the metrics to calculate

• metric_keys (dict of tuples) – the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label will be used for metric calculation

• verbose (bool) – verbosity of the test process

• prepare_batch (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices

• convert_fn (function) – function to convert a batch of tensors to numpy

• **kwargs – additional keyword arguments

Returns

• dict – all predictions obtained by feeding the test_data through the network

• dict – all metrics calculated upon the test_data and the obtained predictions

PyTorchExperiment

class PyTorchExperiment(params: Union[str, delira.training.parameters.Parameters], model_cls: delira.models.abstract_network.AbstractPyTorchNetwork, n_epochs=None, name=None, save_path=None, key_mapping=None, val_score_key=None, optim_builder=<function create_optims_default_pytorch>, checkpoint_freq=1, trainer_cls=<class 'delira.training.pytorch_trainer.PyTorchNetworkTrainer'>, **kwargs)

Bases: delira.training.experiment.BaseExperiment

_resolve_kwargs(kwargs: Optional[dict])

Merges given kwargs with self.kwargs If same argument is present in both kwargs, the one from the given kwargs will be used here

Parameters

kwargs (dict) – the given kwargs to merge with self.kwargs

Returns

merged kwargs

Return type

dict

_resolve_params(params: Optional[delira.training.parameters.Parameters])

Merges the given params with self.params. If the same argument is given in both params, the one from the currently given parameters is used here

Parameters

params (Parameters or None) – the parameters to merge with self.params

Returns

the merged parameter instance

Return type

Parameters

_setup_test(params, model, convert_batch_to_npy_fn, prepare_batch_fn, **kwargs)

Parameters

• params (Parameters) – the parameters containing the model and training kwargs (ignored here, just passed for subclassing and unified API)

• model (AbstractNetwork) – the model to test

• convert_batch_to_npy_fn (function) – function to convert a batch of tensors to numpy

• prepare_batch_fn (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices

• **kwargs – additional keyword arguments

Returns

the created predictor

Return type

Predictor

_setup_training(params, **kwargs)

Handles the setup for training case

Parameters

• params (Parameters) – the parameters containing the model and training kwargs

• **kwargs – additional keyword arguments

Returns

the created trainer

Return type

BaseNetworkTrainer

kfold(data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, num_epochs=None, num_splits=None, shuffle=False, random_seed=None, split_type='random', val_split=0.2, label_key='label', train_kwargs: dict = None, test_kwargs: dict = None, metric_keys: dict = None, params=None, verbose=False, **kwargs)

Performs a k-Fold cross-validation

Parameters

• data (BaseDataManager) – the data to use for training(, validation) and testing. Will be split based on split_type and val_split

• metrics (dict) – dictionary containing the metrics to evaluate during k-fold

• num_epochs (int or None) – number of epochs to train (if not given, will either be extracted from params, self.parms or self.n_epochs)

• num_splits (int or None) – the number of splits to extract from data. If None: uses a default of 10

• shuffle (bool) – whether to shuffle the data before splitting or not (implemented by index-shuffling rather than actual data-shuffling to retain potentially lazy-behavior of datasets)

• random_seed (None) – seed to seed numpy, the splitting functions and the used backend-framework

• split_type (str) – must be one of [‘random’, ‘stratified’] if ‘random’: uses random data splitting if ‘stratified’: uses stratified data splitting. Stratification will be based on label_key

• val_split (float or None) – the fraction of the train data to use as validation set. If None: No validation will be done during training; only testing for each fold after the training is complete

• label_key (str) – the label to use for stratification. Will be ignored unless split_type is ‘stratified’. Default: ‘label’

• train_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the train data. If None: empty dict will be passed

• metric_keys (dict of tuples) – the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label will be used for metric calculation

• test_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the test and validation data. If None: empty dict will be passed

• params (:class:`Parameters`or None) – the training and model parameters (will be merged with self.params)

• verbose (bool) – verbosity

• **kwargs – additional keyword arguments

Returns

• dict – all predictions from all folds

• dict – all metric values from all folds

Raises

ValueError – if split_type is neither ‘random’, nor ‘stratified’

See also

• sklearn.model_selection.KFold

and sklearn.model_selection.ShuffleSplit for random data-splitting

• sklearn.model_selection.StratifiedKFold

and sklearn.model_selection.StratifiedShuffleSplit for stratified data-splitting

• BaseDataManager.update_from_state_dict() for updating the

data managers by kwargs

• BaseExperiment.run() for the training

• BaseExperiment.test() for the testing

Notes

using stratified splits may be slow during split-calculation, since each item must be loaded once to obtain the labels necessary for stratification.

static load(file_name)

Loads whole experiment

Parameters

file_name (str) – file_name to load the experiment from

resume(save_path: str, train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)

Resumes a previous training by passing an explicit save_path instead of generating a new one

Parameters

• save_path (str) – path to previous training

• train_data (BaseDataManager) – the data to use for training

• val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None

• params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)

• **kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

run(train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)

Setup and run training

Parameters

• train_data (BaseDataManager) – the data to use for training

• val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None

• params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)

• **kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

save()

Saves the Whole experiments

setup(params, training=True, **kwargs)

Defines the setup behavior (model, trainer etc.) for training and testing case

Parameters

• params (Parameters) – the parameters to use for setup

• training (bool) – whether to setup for training case or for testing case

• **kwargs – additional keyword arguments

Returns

• BaseNetworkTrainer – the created trainer (if training=True)

• Predictor – the created predictor (if training=False)

See also

• BaseExperiment._setup_training() for training setup

• BaseExperiment._setup_test() for test setup

test(network, test_data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, metric_keys=None, verbose=False, prepare_batch=None, convert_fn=None, **kwargs)

Setup and run testing on a given network

Parameters

• network (AbstractNetwork) – the (trained) network to test

• test_data (BaseDataManager) – the data to use for testing

• metrics (dict) – the metrics to calculate

• metric_keys (dict of tuples) –

  the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label

  will be used for metric calculation

• verbose (bool) – verbosity of the test process

• prepare_batch (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices. If not further specified uses the network’s prepare_batch with CPU devices

• convert_fn (function) – function to convert a batch of tensors to numpy if not specified defaults to convert_torch_tensor_to_npy()

• **kwargs – additional keyword arguments

Returns

• dict – all predictions obtained by feeding the test_data through the network

• dict – all metrics calculated upon the test_data and the obtained predictions

TfExperiment

class TfExperiment(params: Union[str, delira.training.parameters.Parameters], model_cls: delira.models.abstract_network.AbstractTfNetwork, n_epochs=None, name=None, save_path=None, key_mapping=None, val_score_key=None, optim_builder=<function create_optims_default_tf>, checkpoint_freq=1, trainer_cls=<class 'delira.training.tf_trainer.TfNetworkTrainer'>, **kwargs)

Bases: delira.training.experiment.BaseExperiment

_resolve_kwargs(kwargs: Optional[dict])

Merges given kwargs with self.kwargs If same argument is present in both kwargs, the one from the given kwargs will be used here

Parameters

kwargs (dict) – the given kwargs to merge with self.kwargs

Returns

merged kwargs

Return type

dict

_resolve_params(params: Optional[delira.training.parameters.Parameters])

Merges the given params with self.params. If the same argument is given in both params, the one from the currently given parameters is used here

Parameters

params (Parameters or None) – the parameters to merge with self.params

Returns

the merged parameter instance

Return type

Parameters

_setup_test(params, model, convert_batch_to_npy_fn, prepare_batch_fn, **kwargs)

Parameters

• params (Parameters) – the parameters containing the model and training kwargs (ignored here, just passed for subclassing and unified API)

• model (AbstractNetwork) – the model to test

• convert_batch_to_npy_fn (function) – function to convert a batch of tensors to numpy

• prepare_batch_fn (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices

• **kwargs – additional keyword arguments

Returns

the created predictor

Return type

Predictor

_setup_training(params, **kwargs)

Handles the setup for training case

Parameters

• params (Parameters) – the parameters containing the model and training kwargs

• **kwargs – additional keyword arguments

Returns

the created trainer

Return type

BaseNetworkTrainer

kfold(data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, num_epochs=None, num_splits=None, shuffle=False, random_seed=None, split_type='random', val_split=0.2, label_key='label', train_kwargs: dict = None, test_kwargs: dict = None, metric_keys: dict = None, params=None, verbose=False, **kwargs)

Performs a k-Fold cross-validation

Parameters

• data (BaseDataManager) – the data to use for training(, validation) and testing. Will be split based on split_type and val_split

• metrics (dict) – dictionary containing the metrics to evaluate during k-fold

• num_epochs (int or None) – number of epochs to train (if not given, will either be extracted from params, self.parms or self.n_epochs)

• num_splits (int or None) – the number of splits to extract from data. If None: uses a default of 10

• shuffle (bool) – whether to shuffle the data before splitting or not (implemented by index-shuffling rather than actual data-shuffling to retain potentially lazy-behavior of datasets)

• random_seed (None) – seed to seed numpy, the splitting functions and the used backend-framework

• split_type (str) – must be one of [‘random’, ‘stratified’] if ‘random’: uses random data splitting if ‘stratified’: uses stratified data splitting. Stratification will be based on label_key

• val_split (float or None) – the fraction of the train data to use as validation set. If None: No validation will be done during training; only testing for each fold after the training is complete

• label_key (str) – the label to use for stratification. Will be ignored unless split_type is ‘stratified’. Default: ‘label’

• train_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the train data. If None: empty dict will be passed

• metric_keys (dict of tuples) – the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label will be used for metric calculation

• test_kwargs (dict or None) – kwargs to update the behavior of the BaseDataManager containing the test and validation data. If None: empty dict will be passed

• params (:class:`Parameters`or None) – the training and model parameters (will be merged with self.params)

• verbose (bool) – verbosity

• **kwargs – additional keyword arguments

Returns

• dict – all predictions from all folds

• dict – all metric values from all folds

Raises

ValueError – if split_type is neither ‘random’, nor ‘stratified’

See also

• sklearn.model_selection.KFold

and sklearn.model_selection.ShuffleSplit for random data-splitting

• sklearn.model_selection.StratifiedKFold

and sklearn.model_selection.StratifiedShuffleSplit for stratified data-splitting

• BaseDataManager.update_from_state_dict() for updating the

data managers by kwargs

• BaseExperiment.run() for the training

• BaseExperiment.test() for the testing

Notes

using stratified splits may be slow during split-calculation, since each item must be loaded once to obtain the labels necessary for stratification.

static load(file_name)

Loads whole experiment

Parameters

file_name (str) – file_name to load the experiment from

resume(save_path: str, train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)

Resumes a previous training by passing an explicit save_path instead of generating a new one

Parameters

• save_path (str) – path to previous training

• train_data (BaseDataManager) – the data to use for training

• val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None

• params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)

• **kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

run(train_data: delira.data_loading.data_manager.BaseDataManager, val_data: delira.data_loading.data_manager.BaseDataManager = None, params: delira.training.parameters.Parameters = None, **kwargs)

Setup and run training

Parameters

• train_data (BaseDataManager) – the data to use for training

• val_data (BaseDataManager or None) – the data to use for validation (no validation is done if passing None); default: None

• params (Parameters or None) – the parameters to use for training and model instantiation (will be merged with self.params)

• **kwargs – additional keyword arguments

Returns

The trained network returned by the trainer (usually best network)

Return type

AbstractNetwork

See also

BaseNetworkTrainer

save()

Saves the Whole experiments

setup(params, training=True, **kwargs)

Defines the setup behavior (model, trainer etc.) for training and testing case

Parameters

• params (Parameters) – the parameters to use for setup

• training (bool) – whether to setup for training case or for testing case

• **kwargs – additional keyword arguments

Returns

• BaseNetworkTrainer – the created trainer (if training=True)

• Predictor – the created predictor (if training=False)

See also

• BaseExperiment._setup_training() for training setup

• BaseExperiment._setup_test() for test setup

test(network, test_data: delira.data_loading.data_manager.BaseDataManager, metrics: dict, metric_keys=None, verbose=False, prepare_batch=<function TfExperiment.<lambda>>, convert_fn=None, **kwargs)

Setup and run testing on a given network

Parameters

• network (AbstractNetwork) – the (trained) network to test

• test_data (BaseDataManager) – the data to use for testing

• metrics (dict) – the metrics to calculate

• metric_keys (dict of tuples) –

  the batch_dict keys to use for each metric to calculate. Should contain a value for each key in metrics. If no values are given for a key, per default pred and label

  will be used for metric calculation

• verbose (bool) – verbosity of the test process

• prepare_batch (function) – function to convert a batch-dict to a format accepted by the model. This conversion typically includes dtype-conversion, reshaping, wrapping to backend-specific tensors and pushing to correct devices. If not further specified uses the network’s prepare_batch with CPU devices

• convert_fn (function) – function to convert a batch of tensors to numpy if not specified defaults to convert_torch_tensor_to_npy()

• **kwargs – additional keyword arguments

Returns

• dict – all predictions obtained by feeding the test_data through the network

• dict – all metrics calculated upon the test_data and the obtained predictions