feat: 99% credible set validation during study_locus_validation

[d0choa]

Includes:

• Filter credible sets by 95% confidence intervals
• Removes unnecessary filters in other steps

Closes opentargets/issues#3468

[addramir]

We previously discussed 99% CS, not 95% to be more comprehensive. And actually removing it from coloc and everything else.
Moreover, this filter doesn't work for our susie credible sets, because we don't really populate thes columns (we know that these CSs are 99%), both columns is95CredibleSet and is99CredibleSet are Null.

[d0choa]

It's doable, but we must fix a few other problems then. Let's discuss this in person

[DSuveges]

• Removes unnecessary filters in other steps

I like the idea of having one single point in the process where things get dropped. Although pruning the locus object would is radically different from the validation of other datasets (as it would not lead to any new flagged objects and the filtered out tags would not get anywhere unlike invalid studies or study loci. They just disappear.) It would make sense to have it in the validation and would make the resulting datasets consistent across all applications.

@addramir

Moreover, this filter doesn't work for our susie credible sets, because we don't really populate thes columns (we know that these CSs are 99%), both columns is95CredibleSet and is99CredibleSet are Null.

To be honest, I don't really like this inconsistency. Would it be possible to make all cred.set dataset similar? Similarly, if we don't care about different levels of confidence, and would keep everything 99%, we can just drop the column from the schema.

[d0choa]

I need to adjust this PR based on the new decisions described on the ticket opentargets/issues#3468

[d0choa]

Implementing the new decisions described on the ticket opentargets/issues#3468 is pretty simple.

Now, all credible sets are annotated when we try to filter them. That would make all the logic work as long as the locus contains a populated posteriorProbability column. (including SuSie credible sets)

We need to remember that the current PICS results are filtered to 95%, so much of this will not have an effect until we re-run PICS.

[DSuveges]

All makes sense:

• Annotating PICSed credible sets upon creation.
• Filtering method has the logic to do the annotation as well.
• Filtering is happening at the validation step.
• As the dataset is already filtered, coloc doesn't need to apply filter anymore.