Memory issues

[silenus092]

Hi

I have a memory problem. For example, When I use the Mpoxvirus reference sequence to align with a partial genome or specific coding sequence. The program used 62 GB+ of RAM, and the OS killed the process before it finished the alignment.

I've come across https://github.com/smarco/BiWFA-paper which seems to offer a solution with lower memory usage. Additionally, I've noticed the presence of low-memory modes in the code (WFA_wrap.pxd).

Will you plan to integrate these features in the Python interface pyfwa to enable the BiWFA algorithm or select memory strategies, or is there an alternative way? Please feel free to suggest me.

I also changed the heuristic functions (adaptive and Xdrop ), but they didn't help in my case.

[kcleal]

Hi @silenus092,
It would be nice to add BiWFA, for sure. However, I don't have much time to add this. Perhaps in a month or two I might take a look.

[JosephLalli]

I've just implemented this option in my fork here: https://github.com/josephlalli/pywfa

Very experimental, no idea how exposing this option affects accuracy or introduces other complications.

To use, when initializing your WavefrontAligner, specify memory_mode 'biwfa'.

from pywfa import WavefrontAligner

a = WavefrontAligner(memory_mode='biwfa')

[silenus092]

Sorry for the late reply, @JosephLalli Thank you so much for looking into this. I used your version, and it seems to work (at least with Mpoxvirus). I will try it with a bacterial genome that has a much larger genome size.

Also, I use WavefrontAligner with the following settings:

alignment = WavefrontAligner(
    refseq,
    match=match,
    mismatch=mismatch,
    gap_opening=gapopen,
    gap_extension=gapextend,
    span="ends-free",
    distance="affine",
    # pattern_begin_free=,  # commented out
    # pattern_end_free=,    # commented out
    text_begin_free=0,
    text_end_free=0,
    wildcard="N",
    memory_mode="biwfa", 
)

However, when I try to use pattern_begin_free and pattern_end_freeIt, they raise an error and stop the process with the following message.

[WFA] BiWFA ends-free has not been tested properly yet (let me know and I'll do it)


ERROR Error in multiprocessing pool: Worker-0 died unexpectedly # this message came from my parallel computing

I have questions about the proper usage and settings for this, or is this something still under development?

[smarco]

Yes. I implemented it, but never had the time to test it properly.

Regular WFA with end-free works smoothly and has been extensively tested.

In the case of biWFA (memory_mode="biwfa"), it could give problems. If the feature is used/important and you find problems, we can take a look and fix it (if any errors appear).

[silenus092]

Hi @smarco, Thank you, we would really appreciate it if you could take some time to review it. We want to use these pattern_XXXX_free options to get correct semiglobal alignments for partial genomes.