Morphologist bias correction crash.

[Mortenj]

The morphologist pipeline crashes at the T1 Bias Correction step, and I don't understand the error message. I am running on a Ubuntu machine.
Any hint on where to begin with troubleshooting this would be very helpful

ErrorlogMorphologist

Brainvisaerrorlog.png (87.09 KiB) Viewed 32380 times

[riviere]

Hi,
Errors in the segmentation tools are not always easy to interpret, I agreee
You can look at the detailed log in brainvisa ("show log" in the menus, or ctrl-L keys) and inspect the commands outputs and error messages. There will probably be more information there (even if its interpretation may still not be completely straightforward).
Denis

[Mortenj]

Hi Denis

The <C-L> log shows only the same as above.
Investigating further, we have isolated the problem to a segfault in VipT1BiasCorrec:

Code: Select all

[Wed May 11 12:44:51 2016] VipT1BiasCorrec[7688]: segfault at 0 ip 00007fe644150d20 sp 00007ffe78ae3b00 error 4 in ld-2.23.so[7fe644147000+26000]

The system is an Ubuntu 16.04 LTS. We would love to have this working, so anything you could tell us would be great

Cheers,
Rubenv (on behalf of mortenj)

[riviere]

Hi,

I have just tried the binary distribution of BrainVisa/Morphologist on a Ubuntu 16.04 system, and for me it runs.
Are you using the same version as the one I've just tested (brainvisa-4.5-linux64-glibc-2.15) ?

If you don't have more than that in the log, and given the point where it crashes (ld-2.23.so) it seems that it is a problem of library loading at startup. So probably a conflict somewhere between libs provided with brainvisa and possibly some system libs. But if so why do I not have the same crash...

Looking at loaded libraries while VipT1BiasCorrection is running, I see on my system that there are a bunch of system libs that still get loaded whereas they should not (Qt plugins, libtiff...) so maybe this has a link (even if I'm not sure). I have spent a fair amount of time trying to disable these Qt plugins loading in the binary version, but apparently I still have not succeeded. It's a nighmare... Anyway in this non-graphical command Qt is not needed.
Maybe you can try disabling the IO formats supported via Qt.
If you can edit the file <brainvisa_dir>/share/aims-4.5/plugins/aimsdata.plugins, the file looks like this:

Code: Select all

attributes = {
  'release_plugins' : [
    'libaimsgis.so',
    'libaimsvida.so',
    'libaimsdicom.so',
    'libaimsecat.so',
    'libaimsfdf.so',
    'libaimsjpeg.so',
    'libaimstiff.so',
    'libaimsqsqlgraphformat.so',
    'libaimsqtformats.so',
    'libaimsminc.so',
    'libaimsply.so',
    'libaimsgifti.so',
    'libconnect-fiber-clustering-plugin-aims.so'
  ], 
}

You can try to comment out the two plugins involving Qt:

Code: Select all

attributes = {
  'release_plugins' : [
    'libaimsgis.so',
    'libaimsvida.so',
    'libaimsdicom.so',
    'libaimsecat.so',
    'libaimsfdf.so',
    'libaimsjpeg.so',
    'libaimstiff.so',
#    'libaimsqsqlgraphformat.so',
#    'libaimsqtformats.so',
    'libaimsminc.so',
    'libaimsply.so',
    'libaimsgifti.so',
    'libconnect-fiber-clustering-plugin-aims.so'
  ], 
}

and re-run, to see if it changes anything.

Denis

[Mortenj]

Hi Denis

Thanks a lot! It actually did, but then Sulci recognition failed. We decided to attempt running on Ubuntu 12, as that is apparently the system which the package has been built on originally.

We haven't had much success, see the attached screenshot.

How can the pipeline fail in the Talairach transformation step when that step has been deselected and is not even reached?

thing.png (141.24 KiB) Viewed 32338 times

[leka0024]

Hi! Sorry I didn't see this thread before I posted my own thread "Morphologist (pipeline) fails".

I have the exact same errors shown in my log file (Ctrl-L), as those shown in post #1 here.
- AC/PC Or Normalization process succeeds
- T1 Bias Correction process fails

I'm also using Ubuntu 16.04 LTS.

The first time I got the error today, a crash window opened up. But I can't get it to reappear anymore (maybe it just does it once). In that window, it had more detail, probably similar to post #3. I know one of the lines says "VipT1BiasCorrection crashed with SIGSEGV in do_lookup_x()".



To answer the last question in the last post on this thread, I think the problem is that there IS a Talairach transform selected in the first step "Image orientation handling". That's why a red X appears there. If you expand it, you should see that.



Let me see if I can get the crash window to reappear by logging out and signing into a new session. Then I'll try to grab more details from the crash window.

[leka0024]

I could not get the crash window to come back, even though I followed the steps for forcing Apport to work.

I did try to comment out the two libraries suggested in post #4 by Riviere.

The T1 Bias Correction process worked! Although there was an error reported:
------------------------------
!! VIP Error:
Negative or NULL limit value!
------------------------------
------------------------------
!! VIP Error:
Execution aborted in function '(distmap)VipComputeFrontPropagationConnectivityDistanceMap'.
------------------------------


However, the pipeline now fails at "Split Brain Mask". The major error is:

System command exited with non null value : 111

RuntimeError
processes.py (2725) in _processExecution:
result = process.execution( self )
BrainSegmentation.py (137) in execution:
context.system( *command )
neuroProcessesGUI.py (2907) in system:
ret = apply( ExecutionContextGUI.system, (self,) + args, kwargs )
processes.py (2882) in system:
raise RuntimeError( _t_( 'System command exited with non null value : %s' ) % str( ret ) )


And looking at the log outputs, this process gets down to lines called "erosion of size X.X mm" where X.X keeps increasing in size. Eventually it gets big enough that a Warning "VIP Warning: Empty volume in VipGet3DConnex" appears. Then the outputs stop after that.

[Yann Cointepas]

Hi,

I also have the crash of Morphologist* on a Laptop running Ubuntu 16.04. Commenting the two plugins only partially solve the problem. Hopefully, I managed to run Morphologist with sulci recognition by using another build of BrainVISA 4.5.0. The build compiled for Glibc 2.6 under Mandriva 2008 (i.e. the second link in the download page) is working without having to comment plugins.

Regards,

Yann

[leka0024]

Yann,

That's great! I will try that version on Wed and report back the results here.

Also, I will try to use an appropriate MRI scan by that time with correct 3D resolution! (this is in reference to the replies from Riviere on my other thread "Morphologist (pipline) fails")

[Yann Cointepas]

It seem that using build for Glibc 2.6 does not always solve the problem. I know one case where both package were not working.

It is possible that these problems are related to missing dynamic libraries in the package. These libraries are found on the system but, even if they have the same version, they may not be binary compatible and this is the source of many hard to find problems. I have been able to fix partially the Glibc 2.15 package (build on Ubuntu 12.04) by adding some libraries related to Gstreamer (an indirect dependency of Qt). But, I do not know yet why sulci recognition is not working.

[riviere]

Hi,
I have to test that but I will not have time to test on a Ubuntu 16.04 system until at least next week (and if I miss this time slot, it will be end of august).
Denis

[leka0024]

Hi Everyone,

The resident got me a 512x512x192 MRI scan (T1 MPRAGE).


Now the Morphologist pipeline worked much better, but:

1. right at the start of "Trying 1st pass of normalization...", it crashed. Then "Killed", then RuntimeError: "in Anatomy Normalization (using SPM) 1 System command exited with non null value : 137". There was also a crash window for MATLAB. (Maybe this is related to the MRC for SPM?)

It changed to "1st pass failed, changing internal transformation initialization. Retrying normalization after changing initialization..."

The AC/PC/IP numbers were output, then error: "!! VIP Error: Negative or NULL limit value! VIP Error: Execution aborted in function '(distmap) Vip Compute Front Propagation Connectivity Distance Map'

After that, it went back to "Trying 1st pass of normalization..." , then "OK, the image seems to be in the correct orientation." and new numbers output for AC/PC/IP.

2. The pipeline worked successfully for many steps, until "Hemisphere processing -> Left hemisphere -> Sulci Recognition" then FAILED and crashed.

Crash window says: "independent_tag_with_registration.py" has stopped unexpectedly
Details --
exe path: /opt/brainvisa/scripts/sigraph/sulci_recognition/
title: crashed with SIGSEGV in elf_machine_rela()



Yann/Riviere,

I'm going to try downloading the BV 4.5.0 built on Mandriva 2008 with glibc 2.6, and see if this helps.

About the comment "I have been able to fix partially the Glibc 2.15 package (build on Ubuntu 12.04) by adding some libraries related to Gstreamer (an indirect dependency of Qt)." in post #10. Does this mean I would need to download BV 4.5.0 built on Ubuntu 12.04 with glib 2.15 again to take advantage of the updates?




Thanks!

[leka0024]

Hello again,

Morphologist pipeline worked using glibc 2.6 (Mandriva 2008) version on my computer running Ubuntu 16.04. The process finished overall in about 1hour 20mins.

Issues?

1) No MATLAB crash this time, but the same overall issue with normalization. Runs first time, skip unselected Reorient Anatomy, AC/PC/IP output, error: "Negative or NULL limit value!". Then it says error: "Bad argument" and execution aborted.

Back to "Trying 1st pass of normalization...", "OK, the image seems to be in the correct orientation", new AC/PC/IP output.

Then it works after that.


2) There were MANY times (several for every step in the pipeline) that had the following warning message:

"QSqlDatabase: QSQLITE driver not loaded QSqlDatabase: available drivers: QSqlDatabase: an instance of QCoreApplication is required for loading driver plugins"

I didn't see this error when running the glibc 2.15 version


Note: my first installation of BV 4.5.0 (glibc 2.15, Ubuntu 12.04) went into the directory "/opt/brainvisa/". When I installed BV 4.5.0 (glibc 2.6, Mandriva 2008) it went into the directory "/brainvisa-4.5.0/" .

But otherwise the directories look identical. I think I can run either version, still.

And when I look at: /brainvisa-4.5.0/lib/qt-plugins/sqldrivers/ , the directory contains "libsqlite.so" .



Now I will try to view the outputs of Morphologist on my MRI scan in Anatomist.

[leka0024]

Everything seemed to work, when I looked at the scans in the Anatomist viewers from BV "viewers" menu.

I guess I will ignore the two issues I posted above, unless you have any suggestion. Now to figure out how to have the Morphologist pipeline run on 100 subjects automatically for 150hrs straight ...



Thanks all!

[riviere]

Hi,
So the Ubuntu 12 version (glibc 2.15) is built using more recent libraries which seem to need more dependencies, some of them had not been identified. We will try to fix them but it will require a bit of time. In the meantime, if you can use the Mandriva 2008 version (glibc 2.6), it's fine.
Denis