padded_alignment() produces invalid output

[AlexanderDilthey]

Hi,

The padded_alignment() function sometimes produces invalid output, in that the strings that specify the pairwise alignment are not of equal length.

Interestingly, this only happens (on the same alignment) when using CRAM format -- BAM and SAM are fine.

I also get these warnings:

substr outside of string at /home/diltheyat/perl5/lib/perl5/x86_64-linux-thread-multi/Bio/DB/HTS/AlignWrapper.pm line 307.
Use of uninitialized value in concatenation (.) or string at /home/diltheyat/perl5/lib/perl5/x86_64-linux-thread-multi/Bio/DB/HTS/AlignWrapper.pm line 307.

Files to reproduce this error:

Reference genome GRCh38_full_plus_hs38d1_analysis_set_minus_alts.fa: https://gembox.cbcb.umd.edu/shared/graph_hackathon/reference/GRCh38_full_plus_hs38d1_analysis_set_minus_alts.fa

CRAM/SAM with one test alignment: https://www.dropbox.com/s/2dbb413ulfrhpma/testCase.zip?dl=0

Code:

use strict;
use warnings;
use Data::Dumper;
use Bio::DB::HTS;

my $referenceFasta = 'GRCh38_full_plus_hs38d1_analysis_set_minus_alts.fa';
my $sam = Bio::DB::HTS->new(-fasta => $referenceFasta, -bam => 'testCase.cram');

my $alignment_iterator = $sam->features(-iterator => 1);
while(my $alignment = $alignment_iterator->next_seq)
{
my ($ref,$matches,$query) = $alignment->padded_alignment;
unless(length($ref) == length($query))
{
warn Dumper("Alignment length mismatch", length($ref), length($query), $alignment->query->name);
next;
}
}

[rishidev]

Thanks for submitting this. I've been able to replicate the issue with the sample code you have provided, and will investigate further.

[AlexanderDilthey]

Hi @rishidev, any news on that bug by any chance? Solving it would really help dealing with long, e.g. Nanopore, reads.

[rishidev]

Hi @AlexanderDilthey, apologies, I have moved on to a different project now, but this is the last thing I have to fix. The SAM and CRAM behaviour are quite different in the code. I think I've identified where the issue is occurring, but need further clarification to see what the fix is supposed to be.

This seems to be around the CIGAR value. The SAM/BAM contain the sequence themselves so don't have the issue. The CRAM reads reference a CIGAR string with a P in it which is 'a silent deletion from the padded reference'. The padded_alignment call looks like it is returning a padded value but the string is being compared to something that hasn't analysed the P when it's chomping out letters, but I want to go through the spec to confirm that making it do so is the correct decision.

[avullo]

Unfortunately, I'm not able to replicate this, I get (v2.9 with htslib 1.7):

[E::cram_get_ref] Failed to populate reference for id 20
[E::cram_decode_slice] Unable to fetch reference #20 41031278..42297761
[E::cram_next_slice] Failure to decode slice

with 2.7 and/or htslib 1.5 is the same.

@AlexanderDilthey, @rishidev can you confirm you still get the same invalid output and warnings?
@AlexanderDilthey is this still relevant for you?

[AlexanderDilthey]

@avullo Definitely still relevant - I think the pairwise alignment feature is one of the most useful of all! I won't be able to check whether it reproduces with 1.7 immediately, but it's on my list!

[avullo]

Thanks @AlexanderDilthey for letting me know. I'm gonna try with 1.5, but can you tell me your set up (OS/Bio::DB::HTS/htslib version)?

Forgot to mention, #54 is resolved.

[avullo]

From the module's README:
4. "Failed to populate reference for id X" error message

This is probably an indication that the CRAM Reference archive is unavailable.

[keiranmraine]

@avullo, I'm sure you're already aware of this but check REF_PATH and REF_CACHE env values as well as any proxy settings. If they are correct the last thing it could be is the ref server is down at ensembl.

http://www.htslib.org/doc/samtools.html#ENVIRONMENT_VARIABLES

[avullo]

@keiranmraine thanks for the trust but you shouldn't be sure: I'm not (yet) well versed with these tools; I was asked to contribute since I have XS experience ;)

[andrewyatz]

ENA you mean @keiranmraine. Ensembl doesn't host the CRAM reference registry

…

On 7 Mar 2018, at 12:08, Keiran Raine ***@***.***> wrote: @avullo, I'm sure you're already aware of this but check REF_PATH and REF_CACHE env values as well as any proxy settings. If they are correct the last thing it could be is the ref server is down at ensembl. http://www.htslib.org/doc/samtools.html#ENVIRONMENT_VARIABLES — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[keiranmraine]

Sorry yes, ENA at EBI.

[avullo]

Hi @AlexanderDilthey, little but not negligible progress so far.

I and @rishidev managed to isolate the problem and we think fixing would require a reimplementation of the way the reference sequence is reconstructed using the CIGAR and MD strings together (borrowed from the Bio::DB::Sam ancestor module). This is not entirely trivial so bear with me until I properly attack the problem.

But you have a workaround: just instantiate the HTS instance passing the -force_refseq option. This way, it will use the indexed fasta file so you get the correct reference properly aligned to the read.

Let me know if that suits you for your immediate needs. I'm going to address the problem anyway.