KeyError while running Morphologist on command line

[durai2323]

Hello forum members,

I am trying to run the morphologist pipeline on the command line without the bvproc(with normalizaiton enabled). It has 15 arguments.

This is the command used -

Code: Select all

python -m brainvisa.axon.runprocess morphologist \
    t1mri=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/220071_1-t1.nii    \
    perform_normalization=1    \
    anterior_commissure=none    \
    posterior_commissure=none    \ 
interhemispheric_point=none    \
    left_hemisphere_point=none    \
    t1mri_nobias=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/default_analysis/nobias_220071_1-t1.nii    \
    histo_analysis=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/default_analysis/nobias_220071_1-t1.han    \
    split_brain=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/default_analysis/segmentation/voronoi_220071_1-t1.nii    \
    left_graph=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/default_analysis/folds/3.1/L220071_1-t1.arg    \
    right_graph=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/default_analysis/folds/3.1/R220071_1-t1.arg    \
    perform_sulci_recognition=true    \
    left_labelled_graph=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/default_analysis/folds/3.1/default_session_auto/L220071_1-t1_default_session_auto.arg    \
    right_labelled_graph=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/default_analysis/folds/3.1/default_session_auto/R220071_1-t1_default_session_auto.arg  \
    commissure_coordinates=/mnt2/1000BRAINS/subjects/220071_1-t1/t1mri/default_acquisition/220071_1-t1.APC

And I get the following error -

Code: Select all

Loading toolbox cortical_surface
Loading toolbox data management
Loading toolbox datamind
Loading toolbox freesurfer
Loading toolbox morphologist
Loading toolbox spm
Loading toolbox structural_analysis
Loading toolbox tools
Loading toolbox viewers
Loading toolbox My processes
Headless Anatomist running in normal (non-headless) mode
Traceback (most recent call last):
  File "/home/brainvisauser/brainvisa-4.6.0/lib/python2.7/pdb.py", line 1314, in main
    pdb._runscript(mainpyfile)
  File "/home/brainvisauser/brainvisa-4.6.0/lib/python2.7/pdb.py", line 1233, in _runscript
    self.run(statement)
  File "/home/brainvisauser/brainvisa-4.6.0/lib/python2.7/bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "/home/brainvisauser/brainvisa-4.6.0/python/brainvisa/axon/runprocess.py", line 404, in <module>
    process = get_process_with_params(process_name, iterated, *args, **kwargs)
  File "/home/brainvisauser/brainvisa-4.6.0/python/brainvisa/axon/runprocess.py", line 131, in get_process_with_params
    context._setArguments(*(process,) + tuple(args), **kwargs)
  File "/home/brainvisauser/brainvisa-4.6.0/python/brainvisa/processes.py", line 2848, in _setArguments
    proc.setValue(argname, v)
  File "/home/brainvisauser/brainvisa-4.6.0/python/brainvisa/processes.py", line 918, in setValue
    newValue = self.signature[name].findValue(value)
  File "/home/brainvisauser/brainvisa-4.6.0/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'commissure_coordinates'

I tried to debug this error and found that the arguments are parsed into a python dictionary data structure called kwargs, with the parameter name as key and the path/flag as the value. This dict, kwargs, has all the 15 arguemnts successfully stored in it. It is then passed to the morphologist instance of the process class to be stored as an attribute. When it is passed as an attribute to the morphologist instance, only 14 out of 15 arguments are stored and the "commissure_cooridnates" key-value pair is not stored. This is giving rise to the keyerror when the "commissure_cooridnates" key is queried.

Has anybody experienced this error? Do you know what is the issue causing this error?

Thanks,
Durai

[riviere]

Hi Durai,
When I copy/paste the commandline to try it I don't get exactly the same error, but I get a different one (however I'm not using version 4.6.0, I am using the bugfix branch of the sources):

Code: Select all

ValueError: invalid literal for int() with base 10: 'true'

This is due the the boolean value of "perform_sulci_recognition"' which is "true" and should be "True" (or 1) to be recognized by python.
Then with the given parameters, you also need to activate databasing because the normalization steps and sulci recognition steps need some additional data which are found in the internal database of brainvisa, so you wil lneed to add the "--enabledb" after "python -m brainvisa.axon.runprocess".
Now I havent't tried with version 4.6.0 yet to understand why the errors are different, so I am not sure this will completely solve your problem.
Denis

[durai2323]

Then with the given parameters, you also need to activate databasing because the normalization steps and sulci recognition steps need some additional data which are found in the internal database of brainvisa, so you wil lneed to add the "--enabledb" after "python -m brainvisa.axon.runprocess".

Hi Denis,

We have partially solved this problem. Our goal was to run morphologist from the command line. Our current solution is to accomplish this in 3 steps -

1) run the import process

Code: Select all

#!/bin/bash
subj_nifti_prefix=$1
dbase=sbr51
python -m brainvisa.axon.runprocess --enabledb ImportT1MRI \
input=${dbase}/${subj_nifti_prefix}.nii.gz \
output=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/${subj_nifti_prefix}.nii.gz \
referential=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/registration/RawT1-${subj_nifti_prefix}_default_acquisition.referential

2) update the database in the GUI


3) run the morphologist with the "--enabledb" option

Code: Select all

#!/bin/bash
subj_nifti_prefix=$1
dbase=sbr51
python -m brainvisa.axon.runprocess --enabledb morphologist \
t1mri=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/${subj_nifti_prefix}.nii.gz \
perform_normalization=1 \
anterior_commissure=none \
posterior_commissure=none \
interhemispheric_point=none \
left_hemisphere_point=none \
t1mri_nobias=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/default_analysis/nobias_${subj_nifti_prefix}.nii.gz \
histo_analysis=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/default_analysis/nobias_${subj_nifti_prefix}.han \
split_brain=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/default_analysis/segmentation/voronoi_${subj_nifti_prefix}.nii.gz \
left_graph=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/default_analysis/folds/3.1/L${subj_nifti_prefix}.arg  \
right_graph=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/default_analysis/folds/3.1/R${subj_nifti_prefix}.arg \
perform_sulci_recognition=1 \
left_labelled_graph=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/default_analysis/folds/3.1/default_session_auto/L${subj_nifti_prefix}_default_session_auto.arg \
right_labelled_graph=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/default_analysis/folds/3.1/default_session_auto/R${subj_nifti_prefix}_default_session_auto.arg

When we follow these 3 steps it works successfully.
But we have 3 major issues -
a) We are forced to use 2 separate processes - one for import and one for morphologist. Can we not specify the input files directly to morphologist process without having to import it first?
b) We are not able to specify custom locations for the input and output files and are forced to follow the path structure and file naming convention of the database. Can we specify our own file locations?
b) If you run the morphologist immediately after the import process completes successfully you get the "commissure_coordinates" missing error. But this error is removed if you first update the database after running import. Thus we are forced to update the database using GUI after the import (step 1) process, thus forcing us to leave the command line and enter into GUI and then return to command line again (is it possible to update via command line?). Then after updating the database (and using the "--enabledb" option) the morphologist process is able to execute successfully. Is there a way to avoid this?

Thank you,
Durai

[riviere]

Hi Durai,

I am surprised: step 2 should not be needed, and is especially incompatible with a batch execution if it is done using the GUI. There is probably a little bug somewhere but I think some people are using it this way here, so it sounds weird. I will try to take time to investigate.

Anyway there are ways to update databases without the GUI. The process updateDatabases cannot be run this way because it is designed as GUI-only, but a little script will do the job:

Code: Select all

#!/usr/bin/env python
import sys
import os
from brainvisa.axon import processes
processes.initializeProcesses()
from brainvisa.data.neuroHierarchy import databases

databse_dir = os.path.abspath(sys.argv[1])
db = databases.database(database_dir)
# db.clear() # maybe not needed as long as the database is not corrupted
db.update()

(call it with the database directory as argument)
Now that you say it, we should make a non-GUI process for that...

Otherwise, the import step is "almost" required, yes, because:
1) it has to perform some conversions / values rescaling to conform to what morphologist is expecting. We actually thought about including the importation step inside the morphologist pipeline, and, well, never took time to do it (and forgot, actually), but that will be done one day...
2) the files organization and completion systems help a lot to determine output files locations and names. It is possible to bypass it, but you would have about 50 file names to specify, it is painful, and brainvisa will not be able to find the files afterwards.

If you really wish to use morphologist outside of any database, you will have to specify *every* output file name. This is not only the parameters of the main pipeline, because the pipeline is using many sub-processes and sub-pipelines which are using (or generating) files which are not on the main pipeline signature. You can actually specify each of them, by passing parameters like "PrepareSubject.t1mri=/home/myself/mymri.nii", but, well, it will be a pain... The pipeline is clearly not designed to be used this way.

Denis

[riviere]

I have modified the updateDatabases process to allow running it in non-GUI mode:
ftp://ftp.cea.fr/pub/dsv/anatomist/upda ... tabases.py
You have to replace the older one at the same location in your brainvisa installation.
Then you can run:

Code: Select all

python -m brainvisa.axon.runprocess --enabledb updateDatabases /path/to/a/database
# or:
python -m brainvisa.axon.runprocess --enabledb updateDatabases '["/path/to/a/database1", "/path/to/another/database2"]'

Denis

[durai2323]

Hi Denis,

Thank you for modifying updateDatabases.py. I checked if the command line call works and it does! I will get back to you after more testing.

Best,
Durai

[durai2323]

Hi Denis,

Code: Select all

python -m brainvisa.axon.runprocess --enabledb updateDatabases /path/to/a/database

This works perfectly but I encountered a bug unfortunately when running the whole pipeline i.e ImportT1MRI+updateDatabase+morphologist.

This time it is the ImportT1MRI process. It does not create the sqlite database files and the *.minf files unless the database was first created manually using the GUI before running the ImportT1MRI process (This is despite --enabledb option being included in the command line call). And because of this error (which is not reported on the command line. So it took me some time to figure out what was wrong - the ImportT1MRI process was not creating the sqlite database files and the *.minf files) it is actually the updateDatabase that throws the keyerror - saying "KeyError: /path/to/a/database". But the problem is with the ImportT1MRI process.

The only workaround to this is, you have to first manually create a database using the GUI (thus generating the sqlite database files and the *.minf files). Then the ImportT1MRI process is able to create the sqlite and *.minf files for the new folders created during the import process. Then the following processes work successfully.

As you can see this makes things difficult for us as we are forced to go to GUI before running the pipeline in order to create the database.
1) Is this a bug in the ImportT1MRI process?
2) Is there a way to create a database on the commandline?
3) I aslo tried to run updateDatabases (command line version) on an unimported raw folder. It throws the same "KeyError: /path/to/a/database".

Here is the output from ImportT1MRI process (without first creating a database in GUI)
https://pastebin.com/rZURXm6d

Here is the output from ImportT1MRI process (after creating a database in GUI before running ImportT1MRI)
https://pastebin.com/7aB7rM3z

Please note the ls /mnt2 outputs clearly showing lack of *.minf files in case 1.

In both cases the same follwoing command was used -

Code: Select all

python -m brainvisa.axon.runprocess --enabledb ImportT1MRI input=${subj_nifti_dir}/${subj_nifti_file} output=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/${subj_nifti_prefix}.nii.gz refer    ential=${dbase}/subjects/${subj_nifti_prefix}/t1mri/default_acquisition/registration/RawT1-${subj_nifti_prefix}_default_acquisition.referential

Let me know if you need more information.

Thanks,
Durai

[riviere]

Oh I see, yes, it seems so familiar to me now that I forget that it is not obvious to new users, but the general principle of brainvisa is that you should create a database first (one or several), which can be done just once, and then import and process data in this database. But the operation of creating the database has always be done using the GUI and we just didn't think that people would need to use it another way.
Anyway it is possible to create a database in a script, but for now it will be a python script. We could make a process for that...
A python script would look like this:

Code: Select all

#!/usr/bin/env python

from brainvisa.axon import processes
processes.initializeProcesses()
from brainvisa.configuration import neuroConfig
from brainvisa import processes
from brainvisa.data import neuroHierarchy
from soma.wip.application.api import Application
from brainvisa.configuration import databases_configuration
import os

def create_database(database_directory, ontology='brainvisa-3.2.0',
                    allow_ro=False, persistent=False):
    if not os.path.exists(database_directory):
        os.makedirs(database_directory)
    database_settings = neuroConfig.DatabaseSettings(database_directory)
    database = neuroHierarchy.SQLDatabase(
        os.path.join(database_directory, "database.sqlite"),
        database_directory,
        ontology,
        context=processes.defaultContext(),
        settings=database_settings)
    neuroHierarchy.databases.add(database)
    neuroConfig.dataPath.append(database_settings)
    try:
        database.clear(context=processes.defaultContext())
        database.update(context=processes.defaultContext())
    except:
        if not allow_ro:
            raise
    if persistent:
        configuration = Application().configuration
        configuration.databases.fso.append(
            databases_configuration.DatabasesConfiguration.FileSystemOntology(
                database_directory, selected=True, read_only=allow_ro))
        configuration.save(neuroConfig.userOptionFile)
    return database

db = create_database('/tmp/database', persistent=True)

Well, I've done a process doing basically the same thing: ftp://ftp.cea.fr/pub/dsv/anatomist/upda ... atabase.py
You can save it in "<brainvisa_dir>/brainvisa/toolboxes/data management/processes/".

Denis

[durai2323]

Hi Denis,

Thank you for the fix. Sorry I was on vacation. I tested it and it works! But you need to update the database after creating it. That is, before you run import on it. Otherwise it throws an error. So the current way I ma runnign this on the command line is as follows -

1) create database
2) update database
3) import database
4) update database
5) morphologist

There are no problems when you follow this order. But step 2 is necessary otherwise there is an error thrown (but morphologist starts anyway). Please see logs below for details -

Without step 2:
https://pastebin.com/74wDP3j5
With step 2:
https://pastebin.com/wsA3Ci0A

Thanks!

[riviere]

Hi,
You're right, there was an error in the filename of the sqlite database...
I have uploaded an updated version of the createdatabase process:
ftp://ftp.cea.fr/pub/dsv/anatomist/upda ... atabase.py

Denis