Checkpointing issue for cases with pre-inlet + internal viscosity

[mojtabaAmiraslanpour]

Hello,

I was performing some simulations using the pre-inlet capability on of my geometries which is not a straight pipe. Checkpoints don't get saved.

(main) Stats. @ 1000 (0.0001 s):
         # of cells: 1 | # of RBC: 1, PLT: 0
         Velocity  -  max.: 0.00481884 m/s, mean: 0.00117142 m/s, rel. app. viscosity: 0
         Force  -  min.: 0.00242548 pN, max.: 0.0943307 pN (1.47248e-05 lf), mean: 0.0311904 pN
(HemoCell) (Saving Functions) Saving Checkpoint at timestep 1000
[mdc-1057-13-1:38753] *** An error occurred in MPI_Allreduce
[mdc-1057-13-1:38753] *** reported by process [3251765249,2]
[mdc-1057-13-1:38753] *** on communicator MPI_COMM_WORLD
[mdc-1057-13-1:38753] *** MPI_ERR_TRUNCATE: message truncated
[mdc-1057-13-1:38753] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[mdc-1057-13-1:38753] ***    and potentially your MPI job)
[mdc-1057-13-1.rc.usf.edu:38747] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[mdc-1057-13-1.rc.usf.edu:38747] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
Script Execution Time: 36.15 seconds
Date: Wed Oct 26 00:17:29 EDT 2022

I mentioned here you have solved the checkpoint issue for the straight pipe but I can't figure out why is it not working for arbitrary geometry?

Thanks

[mojtabaAmiraslanpour]

Any thoughts on this?
This is very critical for me now as my time and resources are limited.

[christina-rice]

@v1kko and @mojtabaAmiraslanpour
How was the checkpoint issue solved for the straight pipe?

I just recloned the repository, and I am having the checkpoint stop the pipeflow example run (I thought it might be related to the hdf5 problem, but I am not sure).

[mojtabaAmiraslanpour]

Well, I did nothing. It works like a charm for the pipeflow case, but when I add the pre-inlet capability --> Boom.

[mojtabaAmiraslanpour]

Any comments on this issue are much appreciated!!

[mojtabaAmiraslanpour]

I just found out this happens only when I have the pre-inlet domain + internal viscosity feature enabled.

[gzavo]

@mojtabaAmiraslanpour ,

Indeed, seems like the preinlet+other feature combinations break checkpointing atm. We are looking into this.

[mojtabaAmiraslanpour]

This issue seems not to be resolved completely yet. The saving part works fine but reading the checkpoint to continue the run, still gives error:

(HemoCell) (Config) reading files/checkpoint/checkpoint.xml
(HemoCell) (Config) Checkpointed config, deferring the loading of the directories (out,log,checkpoint) until loadCheckpoint is called
[mdc-1057-13-4:92232] *** An error occurred in MPI_Recv
[mdc-1057-13-4:92232] *** reported by process [1847394305,30]
[mdc-1057-13-4:92232] *** on communicator MPI_COMM_WORLD
[mdc-1057-13-4:92232] *** MPI_ERR_TRUNCATE: message truncated
[mdc-1057-13-4:92232] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[mdc-1057-13-4:92232] ***    and potentially your MPI job)
[mdc-1057-13-4.rc.usf.edu:92195] 2 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[mdc-1057-13-4.rc.usf.edu:92195] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

[mojtabaAmiraslanpour]

Adding that the checkpoint capability is working for the cases with the pre-inelt but without interior viscosity. When adding interior viscosity it will just get stuck. I doesn't even get the above error for some reason after I tried it later.

[mojtabaAmiraslanpour]

Any comment is appreciated since I am stuck here. Previously it was told to be solved but the example case doesn't have the interior viscosity enabled.

[mojtabaAmiraslanpour]

Still having the problem. Any updates?

[mojtabaAmiraslanpour]

I am sure there should be a quick fix for this. I haven't been able to find this by myself. Is there anybody there having the same issue and wants to help find it?

[gzavo]

Hi @mojtabaAmiraslanpour ,
Seems like this problem have been solved multiple times somehow? At least I see in the commit history multiple mentions.
I'll check if there is something newer on this in the development repo to merge here.

[mojtabaAmiraslanpour]

Hi @gzavo

That is correct, the last commit is related to this, however, it only solved checkpoint saving but the loading of it, still fails.
Also, note that this works perfectly for a problem with pre-inlet but without interior viscosity enabled.

This has been a huge barrier for me when I am trying to run my large jobs and it would be a great help if you could take a look. I can also send a sample case for you to test.

Best,
Mojtaba