The casa_distro_admin command¶

Creation and publication of casa-distro releases and container images.

Casa_distro is the BrainVISA suite distribution swiss knife. It allows to setup a virtual environment and launch BrainVISA software. See http://brainivsa.info/casa-distro and https://github.com/brainvisa/casa-distro for more information

Version : 3.1.3

usage:

casa_distro_admin <general options> <command> [<command parameters>...]

general optional arguments:

-v, --verbose: Display as much information as possible.
--version: Display casa-distro version number and exit.
-h, --help: Display help message and exit. If used after command name, display only the help of this command.

Common parameters:¶

Most commands accept more or less the same parameters.

Many subcommands need environment selection specifications: see the documentation on how to specify an environment.

base_directory

default= $HOME/casa_distro

Directory where images and environments are stored. This parameter can be passed on the commandline, or set via the CASA_BASE_DIRECTORY environment variable.

Commands:¶

help¶

Print global help or help about a command.

Parameters¶

format: format help text in a given text format. Valid values are “text” (default) for raw text, or rst (RST/sphinx format).
full: if true or yes or 1, display each subcommand parameters documentation in the general help.

singularity_deb¶

Create a Debian package to install Singularity. Perform the whole installation process from a rw system and Singularity source. Then put the result in a *.deb file.

Parameters¶

dockerhub:

default=ubuntu:20.04

Name of the base image system to pull from DockerHub.

output_dir:

default=/casa/host/src/development/casa-distro/5.1/doc/source

Location of the resulting Debian package file.

version:

default=3.8.3

Version of Singularity to use. This must be a valid release version. Go language is not included in the final package.

create_base_image¶

Create a new virtual image

Creating the casa-system image:

For Singularity you need to run these commands in order to create the casa-system image:
```
cd "$CASA_BASE_DIRECTORY"
singularity pull ubuntu-18.04.sif docker://ubuntu:18.04
```
Then you can directly use the ubuntu image as base for the run image. You may build a “system” image as follows, but it’s not needed:
```
casa_distro_admin create_base_image base=ubuntu-18.04.sif \
    type=system image_version=5.0
```
For VirtualBox:
1. Download a Ubuntu image to your CASA base directory, e.g. lubuntu-22.04.1-desktop-amd64.iso
2. Run create_user_image:
```
casa_distro_admin create_user_image \
    container_type=vbox \
    type=system \
    base=lubuntu-22.04.1-desktop-amd64.iso \
    image_version=5.3
```
3. Follow the instructions that are printed in the terminal to install and configure Lubuntu appropriately.

Parameters¶

type

type of image to publish. Either “system” for a base system image, or “run” for an image used in a user environment, or “dev” for a developer image.

name

default=casa-{type}-{image_version}

If given, select environment by its name. It replaces type, distro, branch and system and is shorter to select one.

base

Source file used to build the image. The default value depends on image type and container type.

output

default=/casa/home/casa_distro/{name}.{extension}

File location where the image is created.

container_type

default=singularity

Type of virtual appliance to use. Either “singularity”, “vbox” or “docker”.

image_version

default=

Version (or branch) of the image

verbose

default=True

Print more detailed information if value is yes, true or 1.

cleanup (effective only if container_type=singularity)

default=yes

If “no”, “false” or “0”, do not cleanup after a failure during image building. This may allow to debug a problem after the failure. For instance, with Singularity one can use a command like:

sudo singularity run --writable
/tmp/rootfs-79744fb2-f3a7-11ea-a080-ce9ed5978945 /bin/bash

force (allowed only if container_type=singularity)

default=no

If yes, true or 1, erase existing image without asking any question.

fakeroot (allowed only if container_type=singularity)

default=yes

If yes, true or 1, use singularity –fakeroot for building the images (this is the recommended wby). Otherwise, “sudo singularity” must be used so singularity has root access on the host.

memory (allowed only if container_type=vbox)

default=8192

Size in MiB of memory allocated for virtual machine.

video_memory (allowed only if container_type=vbox)

default=50

Size in MiB of video memory allocated for virtual machine.

disk_size (allowed only if container_type=vbox)

default=131072

For vbox container type only. Size in MiB of maximum disk size of virtual machine.

gui (allowed only if container_type=vbox)

default=no

For vbox container type only. If value is “yes”, “true” or “1”, display VirtualBox window.

publish_base_image¶

Upload an image to BrainVISA web site.

Upload is done with rsync in the following remote directory:

brainvisa@brainvisa.info:/var/www/html/brainvisa.info_download/

This directory location can be customized with the following environment variables:

BRAINVISA_PUBLISH_LOGIN (default=brainvisa)
BRAINVISA_PUBLISH_SERVER (default=brainvisa.info)
BRAINVISA_PUBLISH_DIR (default=/var/www/html/brainvisa.info_download)

Parameters¶

type

type of image to publish. Either “system” for a base system image, or “run” for an image used in a user environment, or “dev” for a developer image.

image

Force usage of a specific virtual image instead of the one defined in the environment configuration.

container_type

default=singularity Type of virtual appliance to use. Either “singularity”, “vbox” or “docker”.

verbose

default=True

Print more detailed information if value is yes, true or 1.

create_user_image¶

Create a “user” image given a development environment. The development environment is selected among existing ones its distro and system or simply by its name. Only development environments using the master branch are considered. This command can perform three steps. Each step can be ignored by setting the corresponding option to “no” :

install: perform an installation of the development environment into its installation directory. This modify the development environment by updating its installation directory.
generate: generate a new image for the development environment. The new image is based on base_image and the installation directory of the development environment is copied into the image in /casa/install.

Parameters¶

version [REQUIRED]

Version of the release to create.

name

default={distro}-{version} Name given to the created image.

base_image

default={base_directory}/casa-run-{image_version}{extension} Name of the “run” image used to generate the new user image

distro

default=None

If given, select environment having the given distro name.

branch

default=None

If given, select environment having the given branch.

system

default=None

If given, select environments having the given system name.

image_version

default=None

If given, select environments having the given image version.

environment_name

If given, select dev environment by its name.

container_type

default=None Type of virtual appliance to use. Either “singularity”, “vbox” or “docker”.

output

default={base_directory}/{name}{extension} Path of the output image.

force

default=no If “yes”, “true” or 1, erase existing image without asking any question.

fakeroot (allowed only if container_type=singularity)

default=yes If yes, true or 1, use singularity –fakeroot for building the images (this is the recommended wby). Otherwise, “sudo singularity” must be used so singularity has root access on the host.

base_directory

default= $HOME/casa_distro

Directory where images and environments are stored. This parameter can be passed on the commandline, or set via the CASA_BASE_DIRECTORY environment variable.

install

default=yes If “true”, “yes” or “1”, perform the installation steps: ‘make install-runtime’, as well as ‘make install-doc’ and ‘make install-test’, depending on the install_doc and install_test parameters. If “false”, “no” or “0”, skip all installation steps

install_doc

default=yes If “true”, “yes” or “1”, run ‘make install-doc’ as part of the install step. If “false”, “no” or “0”, skip this step

install_test

default=yes If “true”, “yes” or “1”, run ‘make install-test’ as part of the install step. If “false”, “no” or “0”, skip this step

install_thirdparty

default=none If “none”, no third-party software is installed in the image. If “all”, all available software will be installed during the generate step. If “default”, a default list of software will be installed. If starting with “file://”, then the thirdparty install list will be read from a JSON file. Other values are understood as a list of software (“spm12-standalone,freesurfer”). Each will be installed from their host system location into /usr/local on the container image. Software location will be searched in a small search list (/usr/local, /drf/local, /i2bm/local). If installed in another location, this location may be passed after a = sign in the software name, ex: spm12-standalone=/opt/spm,freesurfer. If a JSON file is used, then the file syntax is a dictionary, keys are thirdparty software names (“spm12-standalone”), and values are directories. If default locations are expected, then the value may be null.

generate

default=yes If “true”, “yes” or “1”, perform the image creation step. If “false”, “no” or “0”, skip this step

cleanup

default=yes If “false”, “no” or “0”, do NOT clean up the temp image during the generate step after failed build, can be helpful for debugging

zip

default=no If “true”, “yes” or “1”, zip the installed files for an “online” installation.

verbose

default=True

Print more detailed information if value is yes, true or 1.

publish_user_image¶

Upload a “user” image to the BrainVISA web site.

Upload is done with rsync in the following remote directory:

brainvisa@brainvisa.info:/var/www/html/brainvisa.info_download/

This directory location can be customized with the following environment variables:

BRAINVISA_PUBLISH_LOGIN (default=brainvisa)
BRAINVISA_PUBLISH_SERVER (default=brainvisa.info)
BRAINVISA_PUBLISH_DIR (default=/var/www/html/brainvisa.info_download)

Parameters¶

image: Force usage of a specific virtual image instead of the one defined in the environment configuration.

bbi_daily¶

BrainVISA Build infrastructure: daily/nightly automated tests

In BrainVISA Build Infrastructure (BBI), the casa_distro_admin bbi_daily command orchestrates automated builds and tests in a given base directory. The bbi_daily command will run the following steps, while (optionally) logging detailed output to a Jenkins server. See Automated tests with bbi_daily for a complete introduction to automated tests.

Self-update casa_distro using ‘git pull’ and restart itself for the next steps
Select dev environments based on the provided filters (distro, branch, system, image_version, name)
Update casa-dev and casa-run images used by the selected environments to the latest version available from the BrainVISA website
For every selected dev environment, perform the following tasks:
1. bv_maker sources
2. bv_maker configure
3. bv_maker build
4. bv_maker doc
5. perform all tests in the dev environment (running the same commands as bv_maker test)
6. create or update a user image with casa_distro_admin create_user_image (i.e. install the compiled software in a new user image based on the casa-run image, where the software is installed under /casa/install)
7. install that user image into a fresh user environment (reference test data are automatically linked from the dev environment)
8. perform all tests in that user environment

The process can be controlled with the command-line options described below, as well as some configuration keys that can be added to the casa_distro.json of each dev environment:

ctest_options: list of command-line options to pass to ctest on the command-line. ctest is used for listing the tests, so you may pass filtering options to restrict the set of tests being run (e.g. ["-R", "disco"]).
bbi_user_config: an optional dictionary for customizing the creation of a the user image and the user environment, where tests of the user image will be run. This dictionary can contain these keys:
- name: the name of the environment where the user image will be set up. Default: dev_config['name'] + '-userimage'
- directory: the full path where the user environment will be set up. Warning: this directory is erased every time ``bbi_daily`` is run. Default: {base_directory} + name
- image: the full path where the user image will be created. Default: {base_directory} + name + extension
- version: the version string that will be set in the user image, e.g. available as the CASA_VERSION environment variable and as the version metadata key. This value version is passed to time.strftime(). Default: '%Y-%m-%d'.
- setup_commands: list of shell commands that will be run in the newly created user environment. May be used to run post-setup tasks, such as configuring additional files in the home directory of the user (e.g. configuring MATLAB licences…).

Parameters¶

distro

default=None

If given, select environment having the given distro name.

branch

default=None

If given, select environment having the given branch.

system

default=None

If given, select environments having the given system name.

image_version

default=None

If given, select environments having the given image version.

name

default=None

If given, select environment by its name. It replaces type, distro, branch and system and is shorter to select one.

jenkins_server

default = None Base URL of the Jenkins server used to send logs (e.g. https://brainvisa.info/builds). If none is given, logs are written to standard output.

jenkins_auth

default = {base_directory}/jenkins_auth Name of a file containing user name and password (can be a token) to use to contact Jenkins server REST API. The file must have only two lines with login on first line and password on second.

update_casa_distro

default = yes If true, yes or 1, update casa_distro

update_base_images

default = yes Boolean indicating if the update images step must be done

bv_maker_steps

default = sources,configure,build,doc Coma separated list of bv_maker commands to perform on dev environments. May be empty to do nothing.

dev_tests

default = yes Boolean indicating if the tests must be performed on dev environments

update_user_images

default = yes Boolean indicating if images of user environment must be recreated

recreate_user_envs

default = yes Boolean indicating if user environments must be wiped and recreated using singularity –bind <dir>:/casa/setup …

user_tests

default = yes Boolean indicating if the tests must be performed on user environments

install_thirdparty

default = none passed to “create_user_image” when update_user_images is true

base_directory

default= $HOME/casa_distro

Directory where images and environments are stored. This parameter can be passed on the commandline, or set via the CASA_BASE_DIRECTORY environment variable.

verbose

default=None

Print more detailed information if value is yes, true or 1.

local_install¶

Run the installation procedure to create a run or dev image on the local machine. Installation can be don step by step. This command is typically used in a VirtualBox machine to debug image creation scenario.

Parameters¶

type

Type of image to install. Either “run” or “dev”.

action

default=None

If not given, list the possible actions for the selected type and indicate if they were done or not. Can have one of the following values:

"next": perform the nex action not already done
"all": perform all action not already done
coma separated list of acions:
    perform all selected actions even if they were already done

version

default=*

Image version to use for searching an image builder file. This is used as a shell pattern, the default value match any version.

log_file

default=/etc/casa_local_install.log

File where information about steps that have been performed is stored.

user

default=brainvisa

Name of the user account to use for non root commands.

convert_image¶

Convert a virtual image to another container type

Parameters¶

source

Source image file.

container_type

default=docker

Type of virtual appliance to use. Either “singularity”, “vbox” or “docker”.

convert_from

default=None

Convert from this container type. If not specified, take it from the source container metadata.

verbose

default=True

Print more detailed information if value is yes, true or 1.

Images and releases creation¶

A casa-distro release is done by a member of the BrainVISA team using the casa_distro_admin command. This document contains all the steps that are necessary to build and publish all versions of casa-distro images.

Creation a base VirtualBox image¶

The base VirtualBox image is a minimal configuration of a system image downloaded from internet as an *.iso file. It serves as starting point for the building of run and dev images. Once created, the image is published on BrainVISA website in the following URL http://brainvisa.info/casa-distro/casa-<iso_name>.ova where <iso name>̀ is the name of the *.iso file of the distribution. For instance if one uses ubuntu-18.04.4-desktop-amd64.iso, the uploaded image name would be casa-ubuntu-18.04.4-desktop-amd64.ova.

Install VirtualBox version 6 or greater
Download an Ubuntu iso file from internet (any Debian based distro may work but only recent Ubuntu LTS are tested).
Run casa_distro_admin create_system to create an empty VirtualBox image with appropriate base settings (e.g. enough maximum disk size)
Perform Ubuntu minimal installation with an autologin account named “brainvisa” and with password “brainvisa”
Perform system updates and install kernel module creation packages :

sudo apt update
sudo apt upgrade
sudo apt install gcc make perl

Set root password to “brainvisa” (this is necessary to automatically connect to the VM to perform post-install)
Reboot the VM
Download and install VirtualBox guest additions
Shut down the VM
Configure the VM in VirualBox (especially 3D acceleration, processors and memory)

Managing container images¶

Singularity and Docker¶

In a general way, we designed casa-distro first using Docker, and then extended it to Singularity. We use Singularity by actually converting Docker images, so images should be built first using Docker. So Docker is needed to build Singularity images for Casa-distro (this is not needed for regular users, which just download system images, and can use singularity alone, or docker alone).

Thus, to build a singularity image, one needs to use first casa_distro_admin create_docker, then casa_distro_admin create_singularity.

Designing new images¶

create_docker may be used to update and rebuild the images which are proposed inside the casa-distro distribution, but this specific use has a limited benefit since they are normally pre-built and are downloaded from the hubs when needed.

create_docker may prove useful however for an application designer who needs to ship specific or additional system packages in the images. In this situation a new image type needs to be designed.

New image types correspond to subdirectories which will looked for in the following directory trees (when they exist):

<workflow_repository>/share/docker/<image_name>/<system>
$HOME/.config/casa-distro/docker/<image_name>/<system>
$HOME/.casa-distro/docker/<image_name>/<system>
<casa_builtin>/docker/<image_name>/<system>

where:

<workflow_repository> is the main workflow repository either passed via the --repository option, or via the CASA_DEFAULT_REPOSITORY environment variable, or in the default location $HOME/casa_distro
<casa_builtin> if the builtin share directory of casa-distro (or its sources)
<image_name> is a name (or type name) for the image, like the builtin ones casa-test, casa-dev etc.
<system> is the name of the system running inside the docker image (the builtin ones are ubuntu-18.04, ubuntu-16.04, ubuntu-14.04, ubuntu-12.04, centos-7.4, windows-7-32, windows-7-64.

So custom, user-defined images can be added in a personal directory. Such an image definition directory should contain at least two files:

casa_distro_docker.yaml is a Yaml file definig dependencies, name and tags for the image. Ex:

dependencies:
    - ../../casa-dev
image_sources:
  - name: pytorch
    tags:
      - ubuntu-16.04
    visibility: public

a Dockerfile The Dockerfile may (should) be based on another image, in the usual way of building docker images. Thus an existing casa-distro image can be the base for a new one.

Once an image is created with docker, it can be converted to singularity using casa_distro create_singularity.

Creating a modified BrainVISA distribution¶

If you are a toolbox developer and want to share a modified BrainVISA distribution with your additional tools and dependencies in it, here is a recipe to build a singularity/apptainer image for your needs. You will need sudo permission on your host system to do so.

create the modified run image:

sudo apptainer build -s casa-run-5.3-23 casa-run-5.3-23.sif
sudo apptainer run -w casa-run-5.3-23 bash

in this bash, install whatever you need, like (just examples):

apt update
apt install -y whatever
pip3 install something_else
cp -a /home/someone/my_additional_software /usr/local/
ln -s /usr/local/my_additional_software/bin/* /usr/local/bin/

and so on. Then exit the container shell

Optionally you can test the image with an existing dev environment, using:
```
my_environment/bin/bv image=casa-run-5.3-23 bash
```
here, test the software you have installed etc. You can test brainvisa tools with the “bv_env” wrapper which sets up env variables and paths:
```
/casa/host/build/bin/bv_env anatomist
```
When you’re done, exit the container shell
recreate a .sif file:
```
sudo apptainer build casa-run-mod-5.3-23.sif casa-run-5.3-23
```
(the creation of the user image doesn’t seem to work using a sandbox as a source image, so we need to convert it back to .sif)
In order to use the new run image as a base for a new user image, we need to have a .json metadata file. Just copy it from the original .sif one:
```
cp casa-run-5.3-23.sif.json casa-run-mod-5.3-23.sif.json
```
(well to do things right we should generate another ID and update the md5 there, but never mind)
create the user image:
```
# this is not needed after you have done it once
sudo apptainer config fakeroot --add $(id -u -n)

casa_distro_admin create_user_image base_image=$(pwd)/casa-run-mod-5.3-23.sif name=brainvisa-spn image_version=5.3 environment_name=brainvisa-master-ubuntu-22.04 container_type=singularity install_thirdparty="spm12-standalone=/usr/local/spm12-standalone" version=1.0.0
```
note that:
- the new user image will contain brainvisa projects which have been configured and built in your environment (here I use “brainvisa-master-ubuntu-22.04” which is my dev environment), and only them. If you have not all the projects we ship, then your distribution will not contain them.
- in our user images we now include a copy of spm12 standalone, which we specify with the parameter install_thirdparty=”spm12-standalone=/usr/local/spm12-standalone”, which copies an already installed SPM12.

The casa_distro_admin command¶

Common parameters:¶

Commands:¶

help¶

Parameters¶

singularity_deb¶

Parameters¶

create_base_image¶

Parameters¶

publish_base_image¶

Parameters¶

create_user_image¶

Parameters¶

publish_user_image¶

Parameters¶

bbi_daily¶

Parameters¶

local_install¶

Parameters¶

convert_image¶

Parameters¶

Images and releases creation¶

Creation a base VirtualBox image¶

Managing container images¶

Singularity and Docker¶

Designing new images¶

Creating a modified BrainVISA distribution¶

Table of Contents

Previous topic

Next topic

The casa_distro_admin command¶

Common parameters:¶

Commands:¶

help¶

Parameters¶

singularity_deb¶

Parameters¶

create_base_image¶

Parameters¶

publish_base_image¶

Parameters¶

create_user_image¶

Parameters¶

publish_user_image¶

Parameters¶

bbi_daily¶

Parameters¶

local_install¶

Parameters¶

convert_image¶

Parameters¶

Images and releases creation¶

Creation a base VirtualBox image¶

Managing container images¶

Singularity and Docker¶

Sharing images¶

Designing new images¶

Creating a modified BrainVISA distribution¶