capsul.study_config module

Configuration of Capsul and external software operated through capsul.

capsul.study_config.config_utils submodule

Utility functions for configuration

Functions

environment()

capsul.study_config.config_utils.environment(sh_file=None, env={})

Function that return a dictionary containing the environment needed by a program (for instance FSL or FreeSurfer).

In the configuration file, the variable are expected to be defined as ‘VARIABLE_NAME=value’.

Parameters:

• sh_file (str (mandatory)) – the path to the sh script used to set up the environment.
• env (dict (optional, default empty)) – the default environment used to parse the configuration sh file.

Returns:

environment – a dict containing the program configuration.

Return type:

dict

capsul.study_config.memory submodule

Mamory caching. Probably mostly obsolete, this code is not much used now.

Classes

UnMemorizedProcess

MemorizedProcess

CapsulResultEncoder

Memory

Functions

get_process_signature()

has_attribute()

file_fingerprint()

class capsul.study_config.memory.CapsulResultDecoder(*args, **kargs)

Deal with ProcessResult in json.

class capsul.study_config.memory.CapsulResultEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, encoding='utf-8', default=None)

Deal with ProcessResult in json.

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, long, float or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true (the default), all non-ASCII characters in the output are escaped with uXXXX sequences, and the results are str instances consisting of ASCII characters only. If ensure_ascii is False, a result may be a unicode instance. This usually happens if the input contains unicode strings or the encoding parameter is used.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an OverflowError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation. Since the default item separator is ‘, ‘, the output might include trailing whitespace when indent is specified. You can use separators=(‘,’, ‘: ‘) to avoid this.

If specified, separators should be a (item_separator, key_separator) tuple. The default is (‘, ‘, ‘: ‘). To get the most compact JSON representation you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

If encoding is not None, then all input strings will be transformed into unicode using that encoding prior to JSON-encoding. The default is UTF-8.

class capsul.study_config.memory.MemorizedProcess(process, cachedir, timestamp=None, verbose=1)

Callable object decorating a capsul process for caching its return values each time it is called.

All values are cached on the filesystem, in a deep directory structure. Methods are provided to inspect the cache or clean it.

Initialize the MemorizedProcess class.

Parameters:

• process (capsul process) – the process instance to wrap.
• cachedir (string) – the directory in which the computation will be stored
• timestamp (float (optional)) – The reference time from which times in tracing messages are reported.
• callback (callable (optional)) – an optional callable called each time after the function is called.
• verbose (int) – if different from zero, print console messages.

class capsul.study_config.memory.Memory(cachedir)

Memory context to provide caching for processes.

`cachedir`

string – the location for the caching. If None is given, no caching is done.

cache()

clear()

Initialize the Memory class.

Parameters:base_dir (string) – the directory name of the location for the caching.

cache(process, verbose=1)

Create a proxy of the given process in order to only execute the process for input parameters not cached on disk.

Parameters:

• process (capsul process) – the capsul Process to be wrapped and cached.
• verbose (int) – if different from zero, print console messages.

Returns:

proxy_process – the returned object is a MemorizedProcess object, that behaves as a process object, but offers extra methods for cache lookup and management.

Return type:

MemorizedProcess object

Examples

Create a temporary memory folder

>>> from tempfile import mkdtemp
>>> mem = Memory(mkdtemp())

Here we create a callable that can be used to apply an fsl.Merge interface to files

>>> from capsul.process import get_process_instance
>>> nipype_fsl_merge = get_process_instance(
...    "nipype.interfaces.fsl.Merge")
>>> fsl_merge = mem.cache(nipype_fsl_merge)

Now we apply it to a list of files. We need to specify the list of input files and the dimension along which the files should be merged.

>>> results = fsl_merge(in_files=['a.nii', 'b.nii'], dimension='t')

We can retrieve the resulting file from the outputs:

>>> results.outputs._merged_file

clear(skips=None)

Remove all the cache appart from those given to the method input.

Parameters:skips (list) – a list of path to keep during the cache deletion.

class capsul.study_config.memory.UnMemorizedProcess(process, verbose=1)

This class replaces MemorizedProcess when there is no cache. It provides an identical API but does not write anything on disk.

Initialize the UnMemorizedProcess class.

Parameters:

• process (capsul process) – the process instance to wrap.
• verbose (int) – if different from zero, print console messages.

capsul.study_config.memory.file_fingerprint(afile)

Computes the file fingerprint.

Do not consider the file content, just the fingerprint (ie. the mtime, the size and the file location).

Parameters:afile (string) – the file to process. Returns:fingerprint – the file location, mtime and size. Return type:tuple

capsul.study_config.memory.get_process_signature(process, input_parameters)

Generate the process signature.

Parameters:

• process (Process) – a capsul process object
• input_parameters (dict) – the process input_parameters.

Returns:

signature – the process signature.

Return type:

string

capsul.study_config.memory.has_attribute(trait, attribute_name, attribute_value=None, recursive=True)

Checks if a given trait has an attribute and optionally if it is set to particular value.

Parameters:

• trait (Trait) – the input trait object.
• attribute_name (string) – the trait attribute name that will be checked.
• attribute_value (object (optional)) – the trait attribute axpected value.
• recursive (bool (optional, default True)) – check for the attribute in the inner traits.

Returns:

res – True if input given trait has an attribute and optionally if it is set to a particular value.

Return type:

bool

capsul.study_config.memory.tuple_json_encoder(obj)

Encode a tuple in order to save it in json format.

Parameters:obj (object) – a python object to encode. Returns:encobj – the encoded object. Return type:object

capsul.study_config.process_instance submodule

Process instance factory

Functions

is_process()

is_pipeline_node()

get_process_instance()

get_node_class()

get_node_instance()

capsul.study_config.process_instance.get_node_class(node_type)

Get a custom node class from module + class string. The class name is optional if the module contains only one node class. It is OK to pass a Node subclass or a Node instance also.

capsul.study_config.process_instance.get_node_instance(node_type, pipeline, conf_dict=None, name=None, **kwargs)

Get a custom node instance from a module + class name (see get_node_class()) and a configuration dict or Controller. The configuration contains parameters needed to instantiate the node type. Each node class may specify its parameters via its class method configure_node.

Parameters:

• node_type (str or Node subclass or Node instance) – node type to be built. Either a class (Node subclass) or a Node instance (the node will be re-instantiated), or a string describing a module and class.
• pipeline (Pipeline) – pipeline in which the node will be inserted.
• conf_dict (dict or Controller) – configuration dict or Controller defining parameters needed to build the node. The controller should be obtained using the node class’s configure_node() static method, then filled with the desired values. If not given the node is supposed to be built with no parameters, which will not work for every node type.
• kwargs – default values of the node instance parameters.

capsul.study_config.process_instance.get_process_instance(process_or_id, study_config=None, **kwargs)

Return a Process instance given an identifier.

Note that it is convenient to create a process from a StudyConfig instance: StudyConfig.get_process_instance()

The identifier is either:

• a derived Process class.
• a derived Process class instance.
• a Nipype Interface instance.
• a Nipype Interface class.
• a string description of the class <module>.<class>.
• a string description of a function to warp <module>.<function>.
• a string description of a module containing a single process <module>
• a string description of a pipeline <module>.<fname>.xml.
• an XML filename for a pipeline.
• a Python (.py) filename with process name in it: /path/process_source.py#ProcessName.
• a Python (.py) filename for a file containg a single process.

Default values of the process instance are passed as additional parameters.

Parameters:

• process_or_id (instance or class description (mandatory)) – a process/nipype interface instance/class or a string description.
• study_config (StudyConfig instance (optional)) – A Process instance belongs to a StudyConfig framework. If not specified the study_config can be set afterwards.
• kwargs – default values of the process instance parameters.

Returns:

result – an initialized process instance.

Return type:

Process

capsul.study_config.process_instance.is_pipeline_node(item)

Check if the input is an instance of pipeline Node

capsul.study_config.process_instance.is_process(item)

Check if the input item is a process class or function with decorator or XML docstring which makes it seen as a process

capsul.study_config.run submodule

Process and pipeline execution management

Functions

run_process()

capsul.study_config.run.run_process(output_dir, process_instance, generate_logging=False, verbose=0, configuration_dict=None, cachedir=None, **kwargs)

Execute a capsul process in a specific directory.

Parameters:

• output_dir (str (mandatory)) – the folder where the process will write results.
• process_instance (Process (madatory)) – the capsul process we want to execute.
• cachedir (str (optional, default None)) – save in the cache the current process execution. If None, no caching is done.
• generate_logging (bool (optional, default False)) – if True save the log stored in the process after its execution.
• verbose (int) – if different from zero, print console messages.
• configuration_dict (dict (optional)) – configuration dictionary

Returns:

• returncode (ProcessResult) – contains all execution information.
• output_log_file (str) – the path to the process execution log file.

capsul.study_config.study_config submodule

Main StudyConfig class for configuration of Capsul software, directories etc.

Classes

StudyConfig

StudyConfigModule

Functions

default_study_config()

class capsul.study_config.study_config.StudyConfig(study_name=None, init_config=None, modules=None, engine=None, **override_config)

Class to store the study parameters and processing options.

StudyConfig is deprecated and will probably be removed in Capsul 3. Please use CapsulEngine instead and its construction function, capsul_engine() when possible.

This in turn is used to evaluate a Process instance or a Pipeline.

StudyConfig has modules (see BrainVISAConfig, FSLConfig, MatlabConfig, SmartCachingConfig, SomaWorkflowConfig, SPMConfig, FOMConfig). Modules are initialized in the constructor, so their list has to be setup before instantiating StudyConfig. A default modules list is used when no modules are specified: StudyConfig.default_modules

StudyConfig configuration is loaded from a global file and then from a study specific file (based on study_name parameter). The global configuration file name is either in os.environ[‘CAPSUL_CONFIG’] or in “~/.config/capsul/config.json”. The study specific configuration file name is either defined in the global configuration or in “~/.config/capsul/<study_name>/config.json”.

from capsul.api import StudyConfig

study_config = StudyConfig(modules=['SPMConfig', 'FomConfig'])
# or:
study_config = StudyConfig(modules=StudyConfig.default_modules +
['FomConfig'])

input_directory

str – parameter to set the study input directory

output_directory

str – parameter to set the study output directory

generate_logging

bool (default False) – parameter to control the log generation

create_output_directories

bool (default True) – Create parent directories of all output File or Directory before running a process

process_output_directory

bool (default False) – Create a process specific output_directory by appending a subdirectory to output_directory. This subdirectory is named ‘<count>-<name>’ where <count> if self.process_counter and <name> is the name of the process.

run()

reset_process_counter()

set_trait_value()

get_trait()

get_trait_value()

update_study_configuration()

set_study_configuration()

Initilize the StudyConfig class

Parameters:

• study_name (Name of the study to configure. This name is used to) – identify specific configuration for a study.
• init_config (if not None, must contain a dictionary that will be used) – to configure this StudyConfig (instead of reading configuration from configuration files).
• modules (list of string (default self.default_modules)) – the names of configuration module classes that will be included in this study configuration.
• engine (CapsulEngine) – this parameter is temporary, it just helps to handle the transition to capsul.engine.CapsulEngine. Don’t use it in client code.
• override_config (dictionary) – The content of these keyword parameters will be set on the configuration after it has been initialized from configuration files (or from init_config).

add_trait(name, *trait)

Add a new trait.

Parameters:

• name (str (mandatory)) – the trait name.
• trait (traits.api (mandatory)) – a valid trait.

get_configuration_dict()

Returns a json compatible dictionary containing current configuration.

get_iteration_pipeline(pipeline_name, node_name, process_or_id, iterative_plugs=None, do_not_export=None, make_optional=None, **kwargs)

Create a pipeline with an iteration node iterating the given process.

Parameters:

• pipeline_name (str) – pipeline name
• node_name (str) – iteration node name in the pipeline
• process_or_id (process description) – as in get_process_instance()
• iterative_plugs (list (optional)) – passed to Pipeline.add_iterative_process()
• do_not_export (list) – passed to Pipeline.add_iterative_process()
• make_optional (list) – passed to Pipeline.add_iterative_process()

Returns:

pipeline

Return type:

Pipeline instance

get_process_instance(process_or_id, **kwargs)

Return a Process instance given an identifier.

The identifier is either:

• a derived Process class.
• a derived Process class instance.
• a Nipype Interface instance.
• a Nipype Interface class.
• a string description of the class <module>.<class>.
• a string description of a function to warp <module>.<function>.
• a string description of a pipeline <module>.<fname>.xml.
• an XML filename for a pipeline

Default values of the process instance are passed as additional parameters.

Parameters:

• process_or_id (instance or class description (mandatory)) – a process/nipype interface instance/class or a string description.
• kwargs – default values of the process instance parameters.

Returns:

result – an initialized process instance.

Return type:

Process

get_trait(trait_name)

Method to access the ‘trait_name’ study configuration element.

Notes

If the ‘trait_name’ element is not found, return None

Parameters:trait_name (str (mandatory)) – the trait name we want to access Returns:trait – the trait we want to access Return type:trait

get_trait_value(trait_name)

Method to access the value of the ‘trait_name’ study configuration element.

Parameters:trait_name (str (mandatory)) – the trait name we want to modify Returns:value – the trait value we want to access Return type:object

initialize_modules()

Modules initialization, calls initialize_module on each config module. This is not done during module instanciation to allow interactions between modules (e.g. Matlab configuration can influence Nipype configuration). Modules dependencies are taken into account in initialization.

load_module(config_module_name, config)

Load an optional StudyConfig module.

Parameters:

• config_module_name (Name of the module to load (e.g. "FSLConfig")) –
• config (dictionary containing the configuration of the study.) –

read_configuration()

Find the configuration for the current study (whose name is defined in self study_name) and returns a dictionary that is a merge between global options and study specific options.

Global option are taken from environment variable CAPSUL_CONFIG if it is defined, otherwise from “~/.config/capsul/config.json” if it exists.

The configuration for a study can be defined the global configuration if it contains a dictionary in it “studies_config” option and if there is a key corresponding to self.study_name in this dictionary. If the corresponding value is a string, it must be a valid json configuration file name (either absolute or relative to the global configuration file). Otherwise, the corresponding value must be a dictionary containing study specific configuration values. If no study configuration is found from global configuration, then a file named “~/.config/capsul/%s/config.json” (where %s is self.study_name) is used if it exists.

reset_process_counter()

Method to reset the process counter to one.

run(process_or_pipeline, output_directory=None, execute_qc_nodes=True, verbose=0, configuration_dict=None, **kwargs)

Method to execute a process or a pipline in a study configuration

environment.

Depending on the studies_config settings, it may be a sequential run, or a parallel run, which can involve remote execution (through soma- workflow).

Only pipeline nodes can be filtered on the ‘execute_qc_nodes’ attribute.

A valid output directory is exepcted to execute the process or the pepeline without soma-workflow.

Parameters:

• process_or_pipeline (Process or Pipeline instance (mandatory)) – the process or pipeline we want to execute
• output_directory (Directory name (optional)) – the output directory to use for process execution. This replaces self.output_directory but left it unchanged.
• execute_qc_nodes (bool (optional, default False)) – if True execute process nodes that are taged as qualtity control process nodes.
• verbose (int) – if different from zero, print console messages.
• configuration_dict (dict (optional)) – configuration dictionary

save_configuration(file)

Save study configuration as json file.

Parameters:file (file or str (mandatory)) – either a writable opened file or the path to the output json file.

set_study_configuration(new_config)

Method to set the new configuration of the study.

If a study configuration element can’t be updated properly, send an error message to the logger.

Parameters:new_config (ordered dict (mandatory)) – the structure that contain the default study configuration: see the class attributes to build this structure.

set_trait_value(trait_name, trait_value)

Method to set the value of a parameter.

Parameters:

• trait_name (str (mandatory)) – the trait name we want to modify
• trait_value (object (madatory)) – the trait value we want to set

update_study_configuration(json_fname)

Update the study configuration from a json file.

Parameters:json_fname (str (mandatory)) – the path to the output json file.

class capsul.study_config.study_config.StudyConfigModule(study_config, configuration)

StudyConfig module base class (abstract)

initialize_callbacks()

Method called just after the first call to initialize_modules.

initialize_module()

Method called to initialize selected study configuration modules on startup. This method does nothing but can be overriden by modules.

name

The name of a module that can be used in configuration to select modules to load.

capsul.study_config.study_config.default_study_config()

On the first call create a StudyConfig instance with defaut configuration (eventualy reading configuration files). Then returns this instance on all subsequent calls.

capsul.study_config.config_modules submodule

StudyConfig configuration modules implementations

capsul.study_config.config_modules.attributes_config submodule

Attributes completion config module

Classes

AttributesConfig

class capsul.study_config.config_modules.attributes_config.AttributesConfig(study_config, configuration)

Attributes-based completion configuration module for StudyConfig

This module adds the following options (traits) in the StudyConfig object:

attributes_schema_paths: list of str (filenames)

attributes shchema module name

attributes_schemas: dict(str, str)

attributes shchemas names

process_completion: str (default:’builtin’)

process completion model name

path_completion: str

path completion model name

initialize_module()

capsul.study_config.config_modules.brainvisa_config submodule

Configuration module which links with Axon

Classes

BrainVISAConfig

class capsul.study_config.config_modules.brainvisa_config.BrainVISAConfig(study_config, configuration)

Configuration module allowing to use BrainVISA / Axon shared data in Capsul processes.

This module adds the following options (traits) in the StudyConfig object:

shared_directory: str (filename)

Study shared directory

initialize_module()

capsul.study_config.config_modules.fom_config submodule

Config module for File Organization models (FOMs)

Classes

FomConfig

class capsul.study_config.config_modules.fom_config.FomConfig(study_config, configuration)

FOM (File Organization Model) configuration module for StudyConfig

Note

FomConfig needs BrainVISAConfig to be part of StudyConfig modules.

This module adds the following options (traits) in the StudyConfig object:

input_fom: str

input FOM

output_fom: str

output FOM

shared_fom: str

shared data FOM

volumes_format: str

Format used for volumes

meshes_format: str

Format used for meshes

auto_fom: bool (default: True)

Look in all FOMs when a process is not found. Note that auto_fom looks for the first FOM matching the process to get completion for, and does not handle ambiguities. Moreover it brings an overhead (typically 6-7 seconds) the first time it is used since it has to parse all available FOMs.

fom_path: list of directories

list of additional directories where to look for FOMs (in addition to the standard share/foms)

use_fom: bool

Use File Organization Models for file parameters completion’

Methods:

capsul.study_config.config_modules.freesurfer_config submodule

FreeSurfer configuration module

Classes

FreeSurferConfig

class capsul.study_config.config_modules.freesurfer_config.FreeSurferConfig(study_config, configuration)

Class to set up freesurfer configuration.

Parse the ‘SetUpFreeSurfer.sh’ file and update dynamically the system environment.

Initilaize the FreeSurferConfig class.

Parameters:study_config (StudyConfig object) – the study configuration we want to update in order to deal with freesurfer functions.

initialize_module()

Set up FSL environment variables according to current configuration.

capsul.study_config.config_modules.fsl_config submodule

FSL configuration module

Classes

FSLConfig

class capsul.study_config.config_modules.fsl_config.FSLConfig(study_config, configuration)

FSL configuration module

initialize_module()

Set up FSL environment variables according to current configuration.

capsul.study_config.config_modules.matlab_config submodule

Matlab configuration module

Classes

MatlabConfig

class capsul.study_config.config_modules.matlab_config.MatlabConfig(study_config, configuration)

Matlab path configuration

initialize_module()

Set up Matlab environment according to current configuration.

capsul.study_config.config_modules.nipype_config submodule

NiPype configuration module

Classes

NipypeConfig

class capsul.study_config.config_modules.nipype_config.NipypeConfig(study_config, configuration)

Nipype configuration.

In order to use nipype spm, fsl and freesurfer interfaces, we need to configure the nipype module.

Initialize the NipypeConfig class.

initialize_callbacks()

When the ‘use_nipype’ trait changes, configure nipype with the new setting.

initialize_module()

Set up Nipype environment variables according to current configuration.

capsul.study_config.config_modules.smartcaching_config submodule

Process execution cache configuration module

Classes

SmartCachingConfig

class capsul.study_config.config_modules.smartcaching_config.SmartCachingConfig(study_config, configuration)

capsul.study_config.config_modules.somaworkflow_config submodule

Soma-Workflow configuration module

Classes

ResourceController

SomaWorkflowConfig

class capsul.study_config.config_modules.somaworkflow_config.ResourceController

Configuration options for one Soma-Workflow computing resource

queue

str – Jobs queue to be used on the computing resource for w orkflow submissions

transfer_paths

list(str) – list of paths where files have to be transferred by soma-workflow

path_translations

dict(str, (str, str)) – Soma-workflow paths translations mapping: {local_path: (identifier, uuid)}

class capsul.study_config.config_modules.somaworkflow_config.SomaWorkflowConfig(study_config, configuration)

Configuration module for Soma-Workflow

Stores configuration options which are not part of Soma-Workflow own configuration, and used to run workflows from CAPSUL pipelines.

The configuration module may also store connected Soma-Workflow WorkflowController objects to allow monitoring and submiting workflows.

use_soma_workflow

bool – Use soma workflow for the execution

somaworkflow_computing_resource

str – Soma-workflow computing resource to be used to run processing

somaworkflow_config_file

filename – Soma-Workflow configuration file. Default: $HOME/.soma_workflow.cfg

somaworkflow_keep_failed_workflows

bool – Keep failed workflows after pipeline execution through StudyConfig

somaworkflow_keep_succeeded_workflows

bool – Keep succeeded workflows after pipeline execution through StudyConfig

somaworkflow_computing_resources_config

dict(str, ResourceController) – Computing resource config dict, keys are resource ids. Values are ResourceController instances

get_resource_id()

set_computing_resource_password()

get_workflow_controller()

connect_resource()

disconnect_resource()

connect_resource(resource_id=None, force_reconnect=False)

Connect a soma-workflow computing resource.

Sets the current resource to the given resource_id (transformed by get_resource_id() if None or “localhost” is given, for instance).

Parameters:

• resource_id (str (optional)) – resource name, may be None or “localhost”. If None, the current one (study_config.somaworkflow_computing_resource) is used, or the localhost if none is configured.
• force_reconnect (bool (optional)) – if True, if an existing workflow controller is already connected, it will be disconnected (deleted) and a new one will be connected. If False, an existing controller will be reused without reconnection.

Returns:

Return type:

WorkflowController object

disconnect_resource(resource_id=None)

Disconnect a connected WorkflowController and removes it from the internal list

get_resource_id(resource_id=None, set_it=False)

Get a computing resource name according to the (optional) input resource_id and Soma-Workflow configuration.

For instance, None or "localhost" will be transformed to the local host id.

Parameters:

• resource_id (str (optional)) –
• set_it (bool (optional)) – if True, the computed resource id will be set as the current resource in the somaworkflow_computing_resource trait value.

Returns:

Return type:

computed resource id.

get_workflow_controller(resource_id=None)

Get a connected WorkflowController for the given resource

set_computing_resource_password(resource_id, password=None, rsa_key_password=None)

Set credentials for a given computinf resource.

Such credentials are stored in the config object, but will not be written when the config is saved in a file. They are thus non- persistant.

Parameters:

• resource_id (str (optional)) –
• password (str (optional)) –
• rsa_key_password (str (optional)) –

capsul.study_config.config_modules.spm_config submodule

SPM configuration module

Classes

SPMConfig

class capsul.study_config.config_modules.spm_config.SPMConfig(study_config, configuration)

SPM configuration.

There is two ways to configure SPM:

• the first one requires to configure matlab and then to set the spm directory.
• the second one is based on a standalone version of spm and requires to set the spm executable directory.