The bv_maker.cfg configuration file¶
You must create this file in the following directory: $HOME/.brainvisa/bv_maker.cfg
.
In this file you can configure several types of directories:
source directory: A source directory contains the source code of a set of selected projects. This source code will be updated (to reflect the changes that occured on BioProj or GitHub servers) each time you use bv_maker sources. You can define as many source directories as you want. In a source directory configuration you can select any project and combine several versions of the same project.
build directory: A build directory will contain compiled version of the projects. A build directory can contain any project but only one version per project is allowed. You can define as many source directory as you want.
A section may also contain conditional parts. See the Conditional subsections section for details.
General structure and syntax of bv_maker configuration file¶
The bv_maker configuration file is composed of sections delimited by a line enclosed in square brackets: [ ... ]
. Each section contains the definition of a directory (either a source directory, or a build directory), or a “general” section. In this file, blank lines are ignored and spaces at the begining or end of lines are also ignored. It means that you can indent and separate lines as you wish. For instance:
[ source $HOME/brainvisa/source ]
+ standard trunk
+ standard bug_fix
- communication
+ perso/myself/myproject
[ build $HOME/brainvisa/build/bug_fix ]
build_type = Release
make_options = -j8
standard bug_fix $HOME/brainvisa/source
[ build $HOME/brainvisa/build/trunk ]
make_options = -j8
standard trunk $HOME/brainvisa/source
+ $HOME/brainvisa/source/perso/myself/myproject
A line begining with #
or //
is considered as a comment and ignored. Comments cannot be added at the end of a line. For instance the following are valid comments:
# A comment
# Another comment
// This is also a comment !
But the following line is not valid:
[ build $HOME/brainvisa/build/trunk ] # This is a syntax error
Each line in a section contains either a “variable” definition, or some section-specific declarations such as components definitions.
Option Variables¶
A variable declaration looks the following:
variable = value
Some variables may support additive or substractive declarations, once they have been defined a first time:
make_options = -j6
make_options += VERBOSE=1
This possibility depends on the variable type (here a list), but also on the variable itself (which must be understood in bv_maker as additive)
Most variables are strings, but a few are lists: elements are separated by blank spaces.
Some variables are “dictionaries” which may contain several sub-variables. A typical example is the env
variable which contains environment variables to be set during execution of the steps for the given directory section. The syntax is the following:
env = PATH: /home/myself/bin:$PATH, LD_LIBRARY_PATH:/home/myself/lib:$LD_LIBRARY_PATH
or, in additive mode:
env += PATH: /home/myself/bin:$PATH, LD_LIBRARY_PATH=/home/myself/lib:$LD_LIBRARY_PATH
The difference between the two above examples is that the first will override the full set of environment variables defined in bv_maker.cfg, whereas the second will only add / replace 2 entries.
The above syntax is somewhat limited. If needed a “pythonic” syntax is allowed:
env += {'PATH': '/home/myself/bin:$PATH', 'LD_LIBRARY_PATH': '/home/myself/lib:$LD_LIBRARY_PATH'}
Variables substitution¶
Some variables (not all) with string values support environment variables substitution ($VARIABLE
), and / or “python-like” string substitution (%(variable)s
)for a few variables (date
, hostname
, time
, os
, a few more in some sections).
Dictionary variables do support both:
env += PATH: $HOME/bin:$PATH, BRAINVISA_TEST_DIR: $HOME/tests-%(hostname)s-%(date)s
Environment variables may be set by the user, or by the system, prior to running bv_maker, or through the env
variables in bv_maker.cfg
sections. Additionally, bv_maker itself sets a few helpful variables:
NCPU
:number of processor cores in the current machine. Useful to pass build options matching the power of the build machine:
make_options = -j$NCPU
Definition of a general section¶
The general section is optional, and contains variable which are independent from each other sections (or shared accross them).
The general section definition starts with a line with the following syntax:
[ general ]
Option variables are stored in this section using the syntax option = value
. The following options are supported:
env
: environment variables dictionary. Note that theenv
dictionary in the general section is handled a bit differently than the one in the other sections: in other sections variables defined this way are local to the current section, and only passed to the actual environment when a commandline is run (such ascmake
,make
etc.). In the general sections variables are actually set globally to the environment, thus they are available all along the bv_maker session, including within the run of bv_maker: this means that variables which are to be used during path substitutions insidebv_maker.cfg
should be defined here.email_notification_by_default
:ON```or ``OFF
(default). If set toON
, email notification will always be used iffailure_email
orsuccess_email
are provided. Otherwise, the default behavior is to use email notification only when thebv_maker
commandline is invoked with the--email
option.global_status_file
: if this file is specified, a line will be appended to it for each source/build directory. This line will log the build status for the given directory: OK/FAILED, last step executed, directory, start and stop date and time, machine and system. It can be parsed and displayed using the commandbv_show_build_log
.failure_email
: email address where bv_maker outputs are sent in case of failure. If not specified, no email will be sent and bv_maker outputs will be sent to the standard output. One email will be sent for each directory and build step that fail.failure_email_by_project
: dictionary of email addresses, in a project-indexed dictionary (json or python syntax). Adresses can be lists of strings. Ex:failure_email_by_project = {'aims': 'maintainer@aims.org', 'anatomist': ['maintainer@aims.org', 'maintainer@anatomist.org']}
jenkins_build_name
: pattern to Jenkins job name used to submit to a Jenkins dashboard. Only used ifjenkins_server_url
is also set. The pattern here may include replacement strings specified in a python style"%(variable)s"
. Allowed variables are:date
directory_id
:directory_id
specified in the current section, or directory name of the build section (short name without path)hostname
os
project
step
time
jenkins_server_url
: URL of a Jenkins server which can be used to log build and tests logs. The log will be sent to the Jenkins dashboard through client commandline interface as an external job.jenkins_token
: Jenkins API token (or password) for Jenkins serverjenkins_username
: login on Jenkins serversuccess_email
: email address where bv_maker outputs are sent in case of success. If not specified, no email will be sent and bv_maker outputs will be sent to the standard output. One email will be sent for each directory and build step that succeeds.smtp_server
: SMTP (email server) hostname to be used to send emailsfrom_email
: displayed expeditor of sent emails. If not specified, it will be<user>-<hostname>@intra.cea.fr
(the suffix is needed, and is correct for our lab)reply_to_email
: displayed reply email address in sent emails. If not specified,appli@saxifrage.saclay.cea.fr
.
Definition of a source directory¶
A source directory definition section starts with a line with the following syntax:
[ source <directory> ]
where <directory>
is the name of the directory that will be created and whose content will synchronized with selected source directories located in BrainVISA Subversion server. The directory name can contain environment variable substitution by using $VARIABLE_NAME
. For instance, on Unix systems, $HOME/brainvisa
will be replaced by the brainvisa directory located in the user home directory. If the specified directory does not exist, it will be created (as well as parent directories) when the sources will be processed by bv_maker.
The content of the source directory section is composed of a set of rules to select and unselect Subversion directories to copy in the source directory. Each source directory is first associated with an empty list of subdirectories. Then, the configuration file is parsed in order to modify this list. Each line in the source directory section correspond to an action that can modify the list. These actions are executed in the order they are given. It means that you can unselect directories previously selected or the contrary. For instance if one wants to select all components but one, he will make a first action to select all components and a second one to remove the component to ignore. There are three kind of actions that can be done to modify this list of subdirectories. The syntax of the configuration rules corresponding to these actions are described in the following paragraphs.
In the source section, it is also possible to define some option variables, delcared in the syntax option = value
. The following options are supported:
build_condition
: a condition which must be True to allow configure and build steps, otherwise they will be skipped. The condition is evaluated in python language, and is otherwise free: it may typically be used to restrict build to certain systems or hostnames, some dates, etc.cross_compiling_dirs
: dictionary of directories.cross_compiling_dirs
contains toolchain substitutions for source directory. This is used when execution needs different path to access sources (i.e.: in windows cross compilation, for pure python components, it is necessary to access source directories through network shares, instead of NFS mount point). For instance, the following configuration line:i686-w64-mingw32://computer/basedir
will replace the source directory path with the UNC path//computer/basedir
in a build directory that uses the i686-w64-mingw32cross_compiling_prefix
. The network share//computer/basedir
must have been properly configured oncomputer
to be accessible.directory_id
: used in Jenkins notificationrevision_control
:ON
(default) orOFF
. If enabled, revision control systems (svn, git) will be used to update the sources. If OFF, the sources directory will be left as is as a fixed sources tree.default_source_dir
: ? I don’t know… FIXMEdefault_steps
: steps performed for this build directory when bv_maker is invoked without specifying steps (typically justbv_maker
). Defaults to:sources
.env
: environment variables dictionaryignore_git_failure
: don’t stop after the sources step if one or more git repositories cannot be updated in fast-forward mode (which also occurs when working on a non-principal branch). Later steps will thus be performed, but the source step will still be reported as failed.revision_control
:ON
(default) orOFF
. When ON, sources components will be updated using revision control systems (RCS) (svn, git…), and a list of valid components will be generated during the sources step and saved in a file, namedcomponents_sources.json
in the main sources directory. If sources are only local, turningrevision_control
to OFF will avoid using RCS, but will still generate the list of components for building.stderr_file
: file used to redirect the standard error stream of bv_maker when email notification is used. This file is “persistant” and will not be deleted. If not specified, it will be mixed with standard output.stdout_file
: file used to redirect the standard output stream of bv_maker when email notification is used. This file is “persistant” and will not be deleted. If neither it norstderr_file
are specified, then a temporary file will be used, and erased when each step is finished.update_git_remotes
:ON
(default) orOFF
. If ON, all git remotes will be fetched, otherwise only the current active branch in a component will be fast-forwarded. The default value in brainvisa-cmake < 3 used to beOFF
, but this was changed in order to make things clearer/easier and to handle git-lfs projects. The former “light” mode (no fetch + detached branch mode) has been deprecated also: it still works for repositories created usinbg bv_maker 2.x but new repositories are not initialized this way any longer. See Git repositories and bv_maker
Add components to the list¶
+ component_selection version_selection
A line starting with a plus will use Subversion to add some directories from the BrainVISA BioProj repository. The selections of the directories is done by selecting components according to their name and version. Once the components are selected, bv_maker is able to find the corresponding directories in BrainVISA repository. component_selection is used to select a list of components according to their name (see Component selection). It is not mandatory to provide a version_selection. If it is given, it is used to further filter the list of selected components according to their version (see Version selection).
Remove components from the list¶
- component_selection version_selection
A line starting with a minus is has the same syntax as the previous action but removes the selected directories from the list.
Add directories to the list¶
subversion components¶
+ repository_directory local_directory
or:
brainvisa repository_directory local_directory
In order to include some directories that do not correspond to registered BrainVISA components, one can directly give the directory name in repository_directory
. This directory name must be given relatively to the main BrainVISA repository URL: https://bioproj.extra.cea.fr/neurosvn/brainvisa. By default, repository_directory
is also used to define where this directory will be in the source directory. It is not mandatory to provide a value for local_directory. If it is given, it is used instead of repositor_directory to define the directory location relatively to the source directory.
For instance, the following configuration will link the repository directory https://bioproj.extra.cea.fr/neurosvn/brainvisa/perso/myself/myproject with the local directory /home/myself/brainvisa/perso/myself/myproject
.
[ source /home/myself/brainvisa ]
+ perso/myself/myproject
Whereas the following configuration will link the same repository directory with the local directory /home/myself/brainvisa/myproject
.
[ source /home/myself/brainvisa ]
+ perso/myself/myproject myproject
git components¶
See also Git repositories and bv_maker
git https://github.com/neurospin/highres-cortex.git master highres-cortex/master
Definition of a build directory¶
A build directory definition section starts with a line with the following syntax:
[ build <directory> ]
where <directory>
is the name of the directory where the compilation results will be written. As the source directory, the build directory name can contain environment variable substitution.
This section defines the list of components that will be built and their version and the source directory where they can be found. The components and versions are defined as they were in the source directory. It is also possible to remove components from the list with a line beginning with a minus.
Build directories control the following bv_maker steps:
configure: configure build and generate Makefiles using CMake.
build: compile programs and libraries, install files (as symbolic links) in the build directory tree so as to be ready for local execution.
doc: generate documentation for the built components.
testref: run tests in a special mode so as to generate reference data for later tests comparisons.
test: run tests
In the build section, it is also possible to define some build options:
cmake_options
: passed to cmake (ex:-DMY_VARIABLE=dummy
)ctest_options
: passed to ctest in the test step (ex:-j4 -VV -R carto*
)directory_id
: used in Jenkins notificationenv
: environment variables dictionarymake_options
: passed to make (ex:-j8
)build_type
:Debug
,Release
or none (no optimization options)build_condition
: a condition which must be True to allow configure and build steps, otherwise they will be skipped. The condition is evaluated in python language, and is otherwise free: it may typically be used to restrict build to certain systems or hostnames, some dates, etc.clean_build
:ON
orOFF
(default), if set, the build tree will be cleaned of obsolete files before the build step (using the commandbv_clean_build_tree
)clean_config
:ON
orOFF
(default), if set, the build tree will be cleaned of obsolete files before the configuration step (using the commandbv_clean_build_tree
)cross_compiling_prefix
: toolchain name to use for cross-compilation mode. Ifcross_compiling_prefix
is set, bv_maker runs in cross-compilation mode for the build directory (i.e. it adds definitions to tell cmake to work using a specific toolchain initial cache file, a specific toolchain file and the definedcross_compiling_prefix
). Toolchain initial cache file must be found at<brainvisa_cmake_directory>/cmake/toolchain/<cross_compiling_prefix>/init-cache.cmake
. Toolchain file must be found at<brainvisa_cmake_directory>/cmake/toolchain/<cross_compiling_prefix>/toolchain.cmake
. For instance, if<brainvisa_cmake_directory>
is$HOME/brainvisa/source/brainvisa-cmake
andcross_compiling_prefix
isi686-w64-mingw32
, searched files are$HOME/brainvisa/source/brainvisa-cmake/cmake/toolchain/i686-w64-mingw32/init-cache.cmake
and$HOME/brainvisa/source/brainvisa-cmake/cmake/toolchain/i686-w64-mingw32/toolchain.cmake
default_steps
: steps performed for this build directory when bv_maker is invoked without specifying steps (typically justbv_maker
). Defaults to:configure build
, but may also includedoc
andtest
.stderr_file
: file used to redirect the standard error stream of bv_maker when email notification is used. This file is “persistant” and will not be deleted. If not specified, it will be mixed with standard output.stdout_file
: file used to redirect the standard output stream of bv_maker when email notification is used. This file is “persistant” and will not be deleted. If neither it norstderr_file
are specified, then a temporary file will be used, and erased when each step is finished.test_ref_data_dir
: directory where reference data will be written (during testref step) and read (during test step) for comparison.test_run_data_dir
: directory where data will be written during the test step.
Example
[ build $HOME/brainvisa/build/bug_fix ]
build_type = Release
make_options = -j8
standard bug_fix $HOME/brainvisa/source
In the above example, the bug_fix version of standard components which are located in $HOME/brainvisa/source
directory will be compiled in the build directory $HOME/brainvisa/build/bug_fix
in Release
mode with the option -j8
passed to make command (compilation distributed on 8 processors).
Variants of build directories¶
A build directory may also be a python virtualenv directory. To specify it the section type may be virtualenv instead of build:
[ virtualenv <directory> ]
A virtualenv directory will be initialized the first time it is used, and a python virtualenv environment will be installed there. Then it will be used as a build directory in addition. This allows to use pip install
commands within it with a local install, just for this build directory.
Syntax for components selection¶
Components can be selected according to their name and (in some context) to their version. This paragraph explain how to use component_selection and version_selection and gives some examples of their usage.
Information about the components, components groups and versions are extracted from git repository and stored in the following file: https://github.com/brainvisa/brainvisa-cmake/blob/master/python/brainvisa/maker/components_definition.py
Component selection¶
A component_selection is a string that is used to select one or more component according to their name. The following rules are used to transform this string into a list of components:
If component_selection is a group name, all components of this group are selected. At the time of this writing, four groups are defined:
all which contains all known components,
opensource for all open source components
standard containing only standard components of BrainVISA project
anatomist containing Anatomist and its dependencies.
If component_selection is a project name, all components of this project are selected
If component_selection is a component name, only this component is selected
Component selection must be a single pattern (with Unix shell-style wildcards) or two patterns separated by a colon:
If there is only one pattern, all components matching this pattern are selected
If there are two patterns, all components that are in a project matching the first pattern and that are matching the second pattern are selected
Version selection¶
To select the version of a component or a group of component, it is possible
to give the exact version number of a branch (4.0) or a tag (4.0.1)
to use one of the following keywords:
development, trunk: trunk version in svn repository
bug_fix, branch, stable : latest stable version, the higher version number in branches directory of svn repository
tag, latest_release: latest tag version, the higher version number in tags directory of svn repository
branch:n : the nth version in branches directory
tag:n : the nth version in tags directory
Examples of components selection¶
Select all versions of all existing components:
all
Select latest release version of all components:
all tag
Select latest bug fixing branch of open source components:
opensource branch
Select all components in project aims with version 4.0.2:
aims 4.0.2
Select development version of soma-workflow component:
soma-workflow trunk
Select latest bug fixing branch of all components in anatomist project:
anatomist:* bug_fix
Conditional subsections¶
A section of the configuration file may contain conditional parts. This allows to specialize parts of the configuration according to host system, host name, or whatever.
Condition blocks¶
A conditional subsection should be located inside an existing section (sources or build). It follows the syntax:
[ if <expression> ]
<config lines>
...
[ else ]
<other config lines>
[ endif ]
The [ else ]
block is of course optional, and a global section end also ends the conditional section, so the [ endif ]
section may be omitted if it is at the end of the section.
Condition expressions¶
The condition expression may contain substitution variables as in the shape %(variable)s
syntax. The following variables are recognized:
os
date
time
Other variables depend on the configuration of the section itself, which is only done later, so they are not available yet when parsing conditions.
The condition expression is then evaluated in python language (using the eval()
function), thus allows all python language syntax and loaded libraries. The expression result is cast to a boolean value.
Thus a configuration may look like the following:
[ build $HOME/brainvisa/build/bug_fix ]
build_type = Release
[ if gethostname() == 'my_machine' ]
make_options = -j8
[ else ]
make_options = -j2
[ endif ]
standard bug_fix $HOME/brainvisa/source
Examples¶
Warning
TO DO
[ source $HOME/brainvisa/source ]
+ standard trunk
+ standard bug_fix
- communication
+ perso/myself/myproject
[ build $HOME/brainvisa/build/bug_fix ]
build_type = Release
make_options = -j8
standard bug_fix $HOME/brainvisa/source
[ build $HOME/brainvisa/build/trunk ]
make_options = -j8
standard trunk $HOME/brainvisa/source
- connectomist-*
+ $HOME/brainvisa/source/perso/myself/myproject
Compiling using Qt 5¶
Add in the [ build ]
section:
[ build $HOME/brainvisa/build/bug_fix ]
# ...
cmake_options += -DDESIRED_QT_VERSION=5
# ...
Compiling using Python3¶
Add in the [ build ]
section:
[ build $HOME/brainvisa/build/bug_fix ]
# ...
cmake_options += -DPYTHON_EXECUTABLE=/usr/bin/python3
# ...
Under Unix (filesystems supporting symbolic links) , after building, a link to the matching python executable will be found in bin/python
so that the python
command within this build tree will point to python3
and will be used by all python scripts.