How to handle lots of data in batch mode?

[jinju]

Thanks, Yann
I did it on terminal for test.

brainvisa -b -c 'defaultContext().runProcess("ImportT1MRI", input="/home/jinju/t1_icbm.mnc", output = { 'protocol': 'a test', 'subject': 'a ddd', 'acquisition': 'default_acquisition' } )'

It doesn't work.
Message is following this:

Loading toolbox RIC
Loading toolbox brainrat
Loading toolbox connectomist
Loading toolbox cortical_surface
Loading toolbox data management
Loading toolbox datamind
Loading toolbox editors
Loading toolbox fMRI
Loading toolbox morphometry
Loading toolbox structural_analysis
Loading toolbox sulci
Loading toolbox t1mri
Loading toolbox tools
Loading toolbox viewers
Loading toolbox My processes
The log file for this session is /home/jinju/.brainvisa/brainvisa3.log
ERROR [Errno 2] No such file or directory: 't1pipeline07'

--------------------------------------------------------------------------------
IOError
neuroProcesses.py (3087) in addDir:

readProcess( ff, category=category, toolbox=toolbox ) # two arguments : process
fullpath and category (directories separated by /)

neuroProcesses.py (2826) in readProcess:

processModule = imp.load_module( moduleName, fileIn, fileName, moduleDescription
)

bv_pipeline.py (5) in <module>:

defaultContext().runProcess("t1pipeline07", mri="/home/jinju/DB/test/ccc/ccc/t1mri/default_acquisition/ccc.ima")

neuroProcesses.py (2011) in runProcess:

_process = getProcessInstance( _process )

neuroProcesses.py (2648) in getProcessInstance:

if isinstance( processIdClassOrInstance, basestring ) and minfFormat(
processIdClassOrInstance )[ 1 ] == minfHistory:

api.py (98) in minfFormat:

source = BufferAndFile( open( source ) )

ERROR unexpected EOF while parsing (<string>, line 1)

--------------------------------------------------------------------------------
SyntaxError
neuro.py (173) in main:

exec f in globals(), localsStartup

Exception exceptions.AttributeError: "'NoneType' object has no attribute 'currentThread'" in <bound method ThreadSafeSQLiteConnection.__del__ of <soma.databases.sqlite_tools.ThreadSafeSQLiteConnection object at 0xa954b0c>> ignored
Exception exceptions.AttributeError: "'NoneType' object has no attribute 'currentThread'" in <bound method ThreadSafeSQLiteConnection.__del__ of <soma.databases.sqlite_tools.ThreadSafeSQLiteConnection object at 0xaa2d90c>> ignored
Exception exceptions.AttributeError: "'NoneType' object has no attribute 'currentThread'" in <bound method ThreadSafeSQLiteConnection.__del__ of <soma.databases.sqlite_tools.ThreadSafeSQLiteConnection object at 0xa6cbacc>> ignored

what's the problem??

[Dominique Geffroy]

Hi,

The error message doesn't seem to be in relation with the command you passed to brainvisa at startup. I think the error is about a process bv_pipeline.py that you put in your $HOME/.brainvisa/processes directory.

By the way, you also have a syntax error in the your command line, you have mixed simple quotes and double quotes (' and "). You should write this :
brainvisa -b -c 'defaultContext().runProcess("ImportT1MRI", input="/home/jinju/t1_icbm.mnc", output = { "protocol": "a test", "subject": "a ddd", "acquisition": "default_acquisition"} )'
And last advice, maybe you should avoid using space in the protocol and subject names. This will create directory names with space which is not very practical...

Dominique

[jinju]

Hi, Dominique

Thank you for your answer.
I did it as you say but I got some error message.
Could you help me to solve the problem again?

Error message:

[jinju@localhost ~]$ brainvisa -b -c 'defaultContext().runProcess("ImportT1MRI", input="/home/jinju/t1_icbm.mnc", output = { "protocol": "a test", "subject": "a ddd", "acquisition": "default_acquisition"} )'
Loading toolbox RIC
Loading toolbox brainrat
Loading toolbox connectomist
Loading toolbox cortical_surface
Loading toolbox data management
Loading toolbox datamind
Loading toolbox editors
Loading toolbox fMRI
Loading toolbox morphometry
Loading toolbox structural_analysis
Loading toolbox sulci
Loading toolbox t1mri
Loading toolbox tools
Loading toolbox viewers
Loading toolbox My processes
The log file for this session is /home/jinju/.brainvisa/brainvisa3.log
ERROR argument output is mandatory

--------------------------------------------------------------------------------
Exception
neuro.py (173) in main:

exec f in globals(), localsStartup

<string> (1) in <module>:

?

neuroProcesses.py (2013) in runProcess:

apply( self._setArguments, (_process,)+args, kwargs )

neuroProcesses.py (1998) in _setArguments:

_process.checkArguments()

neuroProcesses.py (631) in checkArguments:

o.checkValue( p, getattr( self, p, None ) )

writediskitem.py (18) in checkValue:

Parameter.checkValue( self, name, value )

neuroData.py (95) in checkValue:

raise Exception( HTMLMessage(_t_('argument <em>%s</em> is mandatory') % name) )

Exception exceptions.AttributeError: "'NoneType' object has no attribute 'currentThread'" in <bound method ThreadSafeSQLiteConnection.__del__ of <soma.databases.sqlite_tools.ThreadSafeSQLiteConnection object at 0x9d1fb0c>> ignored
Exception exceptions.AttributeError: "'NoneType' object has no attribute 'currentThread'" in <bound method ThreadSafeSQLiteConnection.__del__ of <soma.databases.sqlite_tools.ThreadSafeSQLiteConnection object at 0x9df890c>> ignored
Exception exceptions.AttributeError: "'NoneType' object has no attribute 'currentThread'" in <bound method ThreadSafeSQLiteConnection.__del__ of <soma.databases.sqlite_tools.ThreadSafeSQLiteConnection object at 0x9a96acc>> ignored

Thanks in advance.

Jin-ju

[Dominique Geffroy]

This error message means that brainvisa cannot generate the file name of the output.
Maybe you haven't configured your brainvisa database ? To do that, go to brainvisa => preferences => databases menu and click on the add button. Then you choose a directory for your database, brainvisa will write output data in this directory.

All this is explained in this documentation : http://brainvisa.info/doc/brainvisa-3.1 ... tml#bv_man
For the explanations on databases, look at this chapter : http://brainvisa.info/doc/brainvisa-3.1 ... tml#N1055D

[riviere]

Hi,
a little precision:

Maybe you haven't configured your brainvisa database ? To do that, go to brainvisa => preferences => databases menu and click on the add button. Then you choose a directory for your database, brainvisa will write output data in this directory.

Don't forget to import the T1 image into your database before running the T1 pipeline: the filename you gave as input "/home/jinju/t1_icbm.mnc" doesn't match the database organization here, so output filenames cannot be generated.
But, as Dominique has pointed out, all this is explained in the docs...
Denis

[f4bry]

Hi All,

I also would like to run the processes in batch mode. In particular: Import T1 MRI, Morphologist 2013 and Sulci Recognition.
I have installed BrainVISA 4.4.0.

Is there a tutorial explaining how to set up these scripts? Or, is there an existing script that I could use as a starting point?

Thank you in advance,
Fab

[riviere]

Hi Fab,

I don't remember tutorials cover this topic, now that you say it...

However it is not so difficult to run brainvisa in batch mode: you can do it

• in commandline mode,
  • either using the "brainvisa -b -c ..." mode explained earlier in this post
  • [*/]
  • or using the newer command "axon-runprocess" (use it with --help to hget help). It's basically the matter of naming a process and providing parameters. If you need parameters completion and databasing, don't forget to use the --enabledb option. If not, you will have to specify every one of the parameters (there may be many...)
• from a python script:

  Code: Select all

  #!/usr/bin/env python
  
  from brainvisa import axon
  from brainvisa.configuration import neuroConfig
  import brainvisa.processes
  
  # config options
  neuroConfig.gui = False
  
  axon.initializeProcesses()
  
  # init is done, now run processing
  brainvisa.processes.defaultContext().runProcess('importT1MRI', input='a_mri.nii.gz', output='/data/database/center/a_subject/t1mri/default_acquisition/a_subject.nii')
  

  If you don't need completion/databasing, you can add:

  Code: Select all

  neuroConfig.fastStart = True

  before initializeProcesses(), it will start faster.

Note that importation processes can generally not completely perform input/output parameters completion, because there is no information in the input file name that allows to find out where the imported data should be written.

Denis

[f4bry]

Hi Denis,

I can run a .bvproc script in this way: 'brainvisa -r file.bvproc'

I still have a couple of questions:
- Is it possible to update the database using a .bvproc script, or other brainvisa command line options?
- Once a subject is analyzed with the T1 pipeline the files are 'locked'. How can I unlock them without using the graphical interface?

Thank you very much for your help,
Fab

[riviere]

Hi,

Databasing is disabled when using brainvisa -r.
You can activate it using python -m brainvisa.axon.runprocess instead:

Code: Select all

python -m brainvisa.axon.runprocess --enabledb <process> [<params>]

Denis

[riviere]

I don't think that the regular morphologist pipeline actually locke files.
Anyway the API allows to unlock an item using the unlockData() method:

Code: Select all

item.unlockData()

Denis