Solve step 2 Non-binary MAT file

[lizimai]

Hi! I am running antrax solve step 2 and encountering the problem that some graphs cannot be load.
I have 6 colonies in each movie. The previous steps: track and classification were all normal.

My command for step 2:

bsub -n 4 -W 4:00 -J "s2_c5s[1-238:1]" -R "rusage[mem=6144]" 'antrax solve /cluster/scratch/galciatore/Nema_3/cam5 --movlist $LSB_JOBINDEX --step 2'

The graph did exists and the size of the graph seems normal, however after solve step 2 the file turns empty.
Graph file before solve step 2 (I duplicated it and saved it in another folder)

(anTraX) [galciatore@eu-login-02 cam5_before_solve]$ ls -l graphs/C5/graph_5_5.mat
-rw-r----- 1 galciatore galciatore-group 41229 Mar 22 12:06 graphs/C5/graph_5_5.mat

Not all colonies failed in the solve step 2, for example colony 1 were finished.

(anTraX) [galciatore@eu-login-02 logs]$ grep -rHnoL "Done" matlab_*.log
matlab_link_across_movies.log
matlab_solve_g_1_c_3.log
matlab_solve_g_1_c_4.log
matlab_solve_g_1_c_5.log
matlab_solve_g_1_c_6.log
matlab_solve_g_2_c_2.log
matlab_solve_g_2_c_4.log
matlab_solve_g_2_c_5.log
matlab_solve_g_2_c_6.log

The error message in matlab_solve_g_1_c_3.log

18:12:09 -I- Loading trgraph from antrax/graphs/C3/graph_26_26.mat
18:12:11 -I- Loading trgraph from antrax/graphs/C3/graph_27_27.mat
18:12:13 -I- Loading trgraph from antrax/graphs/C3/graph_28_28.mat
18:12:17 -I- Loading trgraph from antrax/graphs/C3/graph_29_29.mat
18:12:20 -I- Loading trgraph from antrax/graphs/C3/graph_30_30.mat
18:12:21 -I- Loading trgraph from antrax/graphs/C3/graph_31_31.mat
18:12:22 -I- Loading trgraph from antrax/graphs/C3/graph_32_32.mat
18:12:25 -I- Loading trgraph from antrax/graphs/C3/graph_33_33.mat
18:12:29 -I- Loading trgraph from antrax/graphs/C3/graph_34_34.mat
Error using load
Cannot read file /cluster/scratch/galciatore/Nema_3/cam5/antrax/graphs/C3/graph_34_34.mat.
Error in trgraph.load (line 880)

Error in trhandles/loaddata (line 609)

Error in solve_across_movies (line 70)

Error in antrax_mcr_interface (line 53)
MATLAB:load:cantReadFile

[asafgal]

Hi Zimai, sorry for the delayed reply.
Just to make sure I got your problem right: the mat file is ok after step 1 (did you try loading it manually to matlab?), but while running step 2 the file cannot be loaded and get corrupted?

[lizimai]

Hi Asaf, sorry that I also lost track. Indeed this is the case. I also did try loading it manually to matlab after step 1 and it worked.

[asafgal]

Not sure what can be the issue. My suggestion is to try running step two in debug mode, and slowly converge on the problematic point where the file gets corrupted.

[lizimai]

Hi Asaf, thanks again for your response! Could you tell how to run the step in debug mode? I cannot find it in the documentation.

[janamach]

@lizimai In case it's still relevant, one way to turn on the debug mode is to run the following command in the terminal window before you execute the antrax command:

export ANTRAX_DEBUG_MODE=True