…li (#716)

* feat: expose summary statistics qc to hydra cli

* feat: expose locus breaker clumping step

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.5.2 → v0.5.5](astral-sh/ruff-pre-commit@v0.5.2...v0.5.5)
- [github.com/pre-commit/mirrors-mypy: v1.10.1 → v1.11.0](pre-commit/mirrors-mypy@v1.10.1...v1.11.0)
- [github.com/jsh9/pydoclint: 0.5.4 → 0.5.6](jsh9/pydoclint@0.5.4...0.5.6)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Bumps [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions) from 10.8.1 to 10.9.
- [Release notes](https://github.com/facelessuser/pymdown-extensions/releases)
- [Commits](facelessuser/pymdown-extensions@10.8.1...10.9)

---
updated-dependencies:
- dependency-name: pymdown-extensions
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.17.0 to 0.18.0.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.17.0...0.18.0)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: notebook for locus breaker and susie finemapping benchmark

* fix: won't pass tests with matplotlib imported

* feat: ukb_ppp_fm benchmark notebook

* fix: check for nulls as well as nans

* fix: removing matplotlib

* fix: update notebooks

* chore: re-run ukb ppp notebook

* chore: rerun notebook

* fix: updating config paths and fine-mapping methods

* Update ot_locus_to_gene_train.yaml

* docs: macos fix for some functions

* docs: formatting

* build(deps-dev): bump ruff from 0.5.1 to 0.6.1

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.5.1 to 0.6.1.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.5.1...0.6.1)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix: linting issues associated with ruff 0.6

* chore: fixing imports for notebooks

* chore: removing old notebook

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: Daniel Considine <[email protected]>

* chore: pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.5.5 → v0.6.1](astral-sh/ruff-pre-commit@v0.5.5...v0.6.1)
- [github.com/pre-commit/mirrors-mypy: v1.11.0 → v1.11.1](pre-commit/mirrors-mypy@v1.11.0...v1.11.1)

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.18.0 to 0.19.1.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.18.0...0.19.1)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Ochoa <[email protected]>

Bumps [lxml](https://github.com/lxml/lxml) from 5.2.2 to 5.3.0.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](lxml/lxml@lxml-5.2.2...lxml-5.3.0)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps-dev): bump pre-commit from 3.7.1 to 3.8.0

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 3.7.1 to 3.8.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v3.7.1...v3.8.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix: update lock

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: David Ochoa <[email protected]>

* feat: update variant sources for variant idx and drop rsid mapping

* chore: remove variant_index step from etl dag

* chore: move gnomad annotation from dev to static assets

* chore: change gentropy docker image

* feat(convert_to_vcf): write data partitioned

* feat: add vcf merging step

* chore: assert merge is succesful

* fix: remove non canonical variants from vcfs

* chore: pass vcf types to merging rule

* fix: match vep parser order of columns to variant index schema

* fix: ditch `assign_variant_id` in favour of `hash_long_variant_ids`

* chore: update `test_as_vcf_df_without_variant_id`

* fix: correct test mock data

* chore: pre-commit auto fixes [...]

* chore: merge from dev

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

* feat: finngen r11 study index paths

* feat: simplification of finngen susie finemapping ingestion

* feat: efos in finngne study index

* docs: updated docstring

* fix: pass spark_session not gentropy session to step

* fix: read uploaded file  requires file:///

* fix: parallelize csv from url

* fix: typo in docstring

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Suveges <[email protected]>

* fix: adding carma_tau parameter to susie_finemapper

* fix: changing default

* fix: defaults

* refactor: generalised validation logic

* refactor: generalised validation logic

* fix(airflow): fine-tuning DAG for data validation

* fix(validation): study locus uniqueness fixed

* feat: add invalid/valid dataset generation in validation steps/dag (#734)

* fix: does not belong in this PR

* refactor: to be moved to orchestration repo

* docs: respective docs pages for the steps

* docs: ammend docstrings

* revert: maintain dag for now but it should be removed eventually

* feat: validate study_locus dataset to produce valid or invalid df

* feat: adjust DAG to parametrise dataset validation

* fix: duplicated row

* feat: increase abstraction of Dataset validation of rows

* docs: increase clarity of what the function does

* fix: error message

* revert: unintended change

* test: testing dataset filtering by quality flag

---------

Co-authored-by: DSuveges <[email protected]>

* chore: pre-commit auto fixes [...]

---------

Co-authored-by: David Ochoa <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

* feat(variant index): upating schema and logic

* feat(variant index): vep transcript annotation improved with consequence score + transcript index

* refactor: removed hardcoded schema definition from parser

* fix(test): fixing mock variant index

* fix: schema extraction fixed

* fix: doctest issue resolved

* fix: updating vep command in the DAG

* fix: removing un-used sequence ontology terms

* fix: adding example to sorter method + changing variable names

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix(finngen_r11): preserve all studyIds

Preserve all studyIds, even if EFO mapping is missing, so mapping
between studyIndex and StudyLocus is by studyId column is accurate.

* fix: typo in docstring

Co-authored-by: Irene López Santiago <[email protected]>

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>
Co-authored-by: Irene López Santiago <[email protected]>

* feat: adding finemapping method to studylocusid hash

* chore: updating function in needed files

* test: update doc test

* fix: mistakes

* fix: tidying logic

* chore: adding studyLocusId recalculation to pics finemapping method

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [ipython](https://github.com/ipython/ipython) from 8.26.0 to 8.27.0.
- [Release notes](https://github.com/ipython/ipython/releases)
- [Commits](ipython/ipython@8.26.0...8.27.0)

---
updated-dependencies:
- dependency-name: ipython
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.19.1 to 0.20.0.
- [Release notes](https://github.com/fpgmaas/deptry/releases)
- [Changelog](https://github.com/fpgmaas/deptry/blob/main/CHANGELOG.md)
- [Commits](fpgmaas/deptry@0.19.1...0.20.0)

---
updated-dependencies:
- dependency-name: deptry
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: use mapping instead of enum values in valid_rows

* fix: typos

* fix: swap valid and invalid paths

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Szymon Szyszkowski <[email protected]>

* refactor: move VA preparation logic into a separate module

* refactor: generalise per-chromosome summary stats ingestion

* Update src/gentropy/common/per_chromosome.py

Co-authored-by: Szymon Szyszkowski <[email protected]>

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* refactor: move harmonisation into a separate module

* refactor: make column names configurable

* feat: make INFO and A1FREQ columns optional

* docs: expand comments on tmp_variant_annotation_path and variant types

* feat: implement FinnGen UKB meta-analysis ingestion and harmonisation

* chore: remove ot_finngen_ukb_meta.yaml

* chore: remove raw_study_index_path to raw_study_index_path_from_tsv

* fix: use session.write_mode

* style: rename class to FinngenUkbMetaIngestionStep

* feat(validation): adding logic to validate credible sets against variant index

* fix: tidying docstrings

* refactor: drop gentropy config

* feat(airflow): removal of airflow orchestration

BREAKING CHANGE: see https://github.com/opentargets/orchestration

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

Bumps [wandb](https://github.com/wandb/wandb) from 0.17.2 to 0.18.0.
- [Release notes](https://github.com/wandb/wandb/releases)
- [Changelog](https://github.com/wandb/wandb/blob/main/CHANGELOG.md)
- [Commits](wandb/wandb@v0.17.2...v0.18.0)

---
updated-dependencies:
- dependency-name: wandb
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: prevent multiple filters to override spark plan

* feat: mhc quality control flag

* fix: prevent multiple filters to override spark plan

* Revert "fix: prevent multiple filters to override spark plan"

This reverts commit a358781.

* revert: wrong commit

* fix: missing changes due to git chaos

* chore: merge dev toml, update lock

* chore: update lock after upgrading poetry to 1.8.3 v 2

---------

Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: prevent multiple filters to override spark plan

* feat: mhc quality control flag

* fix: typo

* docs: genomic region added to docs

* docs: info added to common index (and title)

* docs: fix paths

* Update src/gentropy/study_locus_validation.py

* fix: remove unnecessary comment

---------
Closes opentargets/issues#3469
Co-authored-by: Szymon Szyszkowski <[email protected]>

* fix: turning OR to beta harmonisation on for GWAS Catalog

* fix(ingestion): addressing beta harmonisation bug

* fix(ingestion): addressing beta harmonisation bug

* test: adding test for effect harmonisation

* test: adding doctests for beta harmonisation

* fix: fixing value checking bug: effect value needs to be cast to double type

* fix: typos in gwas ingestion script

* perf: cache Docker image layers

* chore: temporary add branch to test

* perf: separate AMD and ARM builds for gentropy

* perf: temporarily test only VEP image separately

* perf: test ARM/AMD gentropy/VEP separately

* perf: introduce quick build for all commits

* perf: quick build for all tags

* fix: use correct syntax for tags

* fix: revert accidental changes for the VEP step

Co-authored-by: Szymon Szyszkowski <[email protected]>

Co-authored-by: Daniel Suveges <[email protected]>

* fix(ld clumping): a revised logic allows a more accurate identification of linked loci

* test: adding some test

* chore: pre-commit auto fixes [...]

* test: adding more test cases

* fix: fixing test data

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* refactor(L2GFeatureMatrix): remove schema validation

* refactor(FeatureFactory): reshape feature generation WIP

* chore: pre-commit auto fixes [...]

* chore: set l2gfeature properties with decorator

* chore(l2gfeature): make credible_set and input_dependency instance attributes

* chore(l2gfeature): make credible_set and input_dependency instance attributes

* chore(featurefactory): distanceTssMeanFeature working

* refactor(l2g): improve step dependency management

* feat: implement

* chore: fix mypy issues

* feat: l2gfeaturematrix.from_features_list working

* chore: comment out obsolete refs

* chore(L2GFeatureMatrix): change `mode` attribute to `with_gold_standard`

* refactor(l2g): move feature matrix writing to training module

* feat(L2GFeatureMatrix): accept L2GGoldStandard or StudyLocus as inputs

* feat: implement methods to build a feature matrix based on a studylocus/L2GGoldStandard instance

* feat: coloc logic prototype

* feat(l2g): filter non gwas credible sets at the start of the step

* feat: rewrite colocalisation feature factory

* test: add `test_colocalisation_feature_type`

* test(colocalisation): add test_extract_maximum_coloc_probability_per_region_and_gene

* feat(L2GFeatureInputLoader): support multiple deps by passing loader as kwarg

* test: add integration tests `test_build_feature_matrix`

* chore: drop config yamls

* refactor: move feature classes to datasets module

* docs: update feature docs

* refactor(colocalisation): cleaner joins in `append_right_study_metadata`

* chore: better logging abstract methods

* test: add `L2GFeatureMatrix.test_from_features_list` unit tests

* fix: add goldStandardSet when a gs instance is passed to `from_features_list`

* fix: lowercase colocalisation type and add semantic test

* test: add semantic test for `append_right_study_metadata`

* feat(colocalisation): make `append_right_study_metadata` extensible to left metadata

* fix(colocalisation): append_study_metadata cant take a gold standard

* fix(colocalisation): extract_maximum_coloc_probability_per_region_and_gene cant take a gold standard

* feat: add `StudyLocus` as a dependency of colocalisation features

* fix: add studylocus to input loader in test

* fix: add studylocus to input loader in test

* fix: add studylocus to input loader in test

…yLocusOverlap`) (#782)

* feat: add studyType to StudyLocus schema

* feat: add annotate_study_type function to add studyType to StudyLocus

* fix: remove lines for retrieving studyType as StudyLocus now contains studyType

* fix: add studyType to test input data as StudyLocus now contains studyType

* feat: add leftStudyType and rightStudyType to Colocalisation and StudyLocusOverlap schemas

* feat: update _convert_to_square_matrix and its test with leftStudyType and rightStudyType

* feat: update test_find_overlaps_semantic inputs with leftStudyType and rightStudyType

* feat: update tests in test_colocalisation_method.py with leftStudyType and rightStudyType

* feat: add leftStudyType and rightStudyType when creating StudyLocusOverlap

* feat: add leftStudyType and rightStudyType to Colocalisation results

* fix: remove redundant study_index parameter from filter_by_study_type function def and calls

* fix: remove redundant study_index parameter from find_overlaps function def and calls

* fix: remove leftStudyType from Colocalisation (not needed as always gwas)

* fix: remove leftStudyType from StudyLocusOverlap (not needed as always gwas)

* fix: missing comma

* feat: update tests in test_locus_to_gene.py with studyType and rightStudyType

* feat: update tests (colocalisation, l2g, l2g feature matrix) with rightStudyType

* fix: remove studyType from metadata_cols in append_study_metadata function call

* feat: adding window based clumping to locus

* fix: reverting some changes

* chore: pre-commit auto fixes [...]

* fix: fixing probem introduced by merge conflict

* fix: addressing review comment

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Initial commit of biosample index

* Make minimal class

* Tidy up first draft of adding biosample index

* Add beginning of logic for checking if biosample from a studyindex is in biosample index

* Make early file for merging multiple biosample indices into one

* Finish adding basic iteration of biosample index, needs debugging

* Tweak slightly

* Modified the parser to accept JSON files

* Update biosample index

* Tests and docs

* Updating tests

* Revert GWAS catalog file

* fix(biosample index): update to match pre-commit standards

* fix(biosample index): merging indices fix

* fix(biosample index): update study index qc logic

* fix(biosample index): fix missing mock_biosample_index

* chore(biosample index): change datasource name from ontologies

* fix(biosample index): add dataset doc

* fix(biosample index): change dbXrefs to xrefs

* chore (biosample index): better commenting

Co-authored-by: Daniel Suveges <[email protected]>

* fix(biosample index): various minor tweaks to biosample index

* fix(biosample index): minor bug

* fix(biosample index): fix merge shift to method

* feat(biosample index): make biosampleName not nullable

---------

Co-authored-by: Daniel Suveges <[email protected]>

* feat: study locus validation filters for 95% credible sets

* revert: no longer needed to filter for credible set interval

* feat: annotate credible sets before filter them

* docs: adding more context here