ProteinGym v1.2 release

Author: pascalnotin

README.md

@@ -1,6 +1,6 @@
 # ProteinGym
 
-[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.13936340.svg)](https://doi.org/10.5281/zenodo.13936340)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.14997691.svg)](https://doi.org/10.5281/zenodo.14997691)
 [![PyPI version](https://badge.fury.io/py/proteingym.svg)](https://badge.fury.io/py/proteingym)
 [![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
@@ -96,17 +96,16 @@ ESCOTT | MSA & Structure | [Mustafa Tekpinar, Laurent David, Thomas Henry, Aless
 VenusREM | MSA & Structure | [Yang Tan, Ruilin Wang, Banghao Wu, Liang Hong, Bingxin Zhou. (2024). Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model. ArXiv, abs/2410.21127.](https://arxiv.org/abs/2410.21127)
 RSALOR | MSA & Structure | [Matsvei Tsishyn, Pauline Hermans, Fabrizio Pucci, Marianne Rooman. (2025). Residue conservation and solvent accessibility are (almost) all you need for predicting mutational effects in proteins. bioRxiv.](https://www.biorxiv.org/content/10.1101/2025.02.03.636212v1)
 S3F | Single sequence & Structure | [Zuobai Zhang, Pascal Notin, Yining Huang, Aurelie C. Lozano, Vijil Chenthamarakshan, Debora Marks, Payel Das, Jian Tang. (2024). Multi-Scale Representation Learning for Protein Fitness Prediction. NeurIPS](https://papers.nips.cc/paper_files/paper/2024/hash/b7d795e655c1463d7299688d489e8ef4-Abstract-Conference.html)
-
-Except for the WaveNet model (which only uses alignments to recover a set of homologous protein sequences to train on, but then trains on non-aligned sequences), all alignment-based methods are unable to score indels given the fixed coordinate system they are trained on. Similarly, the masked-marginals procedure to generate the masked-marginals for ESM-1v and MSA Transformer requires the position to exist in the wild-type sequence. All the other model architectures listed above (eg., Tranception, RITA, ProGen2) are included in the indel benchmark.
+SiteRM | MSA | [Sebastian Prillo, Wilson Wu, Yun Song. (2024). Ultrafast classical phylogenetic method beats large protein language models on variant effect prediction. NeurIPS.](https://papers.nips.cc/paper_files/paper/2024/hash/eb2f4fb51ac3b8dc4aac9cf71b0e7799-Abstract-Conference.html)
 
 For clinical baselines, we used dbNSFP 4.4a as detailed in the manuscript appendix (and in `proteingym/clinical_benchmark_notebooks/clinical_subs_processing.ipynb`).
 
 ## Resources
 
-To download and unzip the data, use the following template, replacing {VERSION} with the desired version number (e.g., "v1.1") and {FILENAME} with the specific file you want to download, as listed in the table below. The latest version is v1.1.
+To download and unzip the data, use the following template, replacing {VERSION} with the desired version number (e.g., "v1.2") and {FILENAME} with the specific file you want to download, as listed in the table below. The latest version is v1.2.
 For example, you can download & unzip the zero-shot predictions for all baselines for all DMS substitution assays as follows:
 ```
-VERSION="v1.1"
+VERSION="v1.2"
 FILENAME="DMS_ProteinGym_substitutions.zip"
 curl -o ${FILENAME} https://marks.hms.harvard.edu/proteingym/ProteinGym_${VERSION}/${FILENAME}
 unzip ${FILENAME} && rm ${FILENAME}
@@ -116,10 +115,10 @@ Data | Size (unzipped) | Filename
 --- | --- | --- |
 DMS benchmark - Substitutions | 1.0GB | DMS_ProteinGym_substitutions.zip
 DMS benchmark - Indels | 200MB | DMS_ProteinGym_indels.zip
-Zero-shot DMS Model scores - Substitutions | 31GB | zero_shot_substitutions_scores.zip
-Zero-shot DMS Model scores - Indels | 5.2GB | zero_shot_indels_scores.zip
-Supervised DMS Model performance - Substitutions | 2.7MB | DMS_supervised_substitutions_scores.zip
-Supervised DMS Model performance - Indels | 0.9MB | DMS_supervised_indels_scores.zip
+Zero-shot DMS Model scores - Substitutions | 4.4GB | zero_shot_substitutions_scores.zip
+Zero-shot DMS Model scores - Indels | 313MB | zero_shot_indels_scores.zip
+Supervised DMS Model scores - Substitutions | 3.3GB | DMS_supervised_substitutions_scores.zip
+Supervised DMS Model scores - Indels | 215MB | DMS_supervised_indels_scores.zip
 Multiple Sequence Alignments (MSAs) for DMS assays | 5.2GB | DMS_msa_files.zip
 Redundancy-based sequence weights for DMS assays | 200MB | DMS_msa_weights.zip
 Predicted 3D structures from inverse-folding models | 84MB | ProteinGym_AF2_structures.zip
@@ -209,15 +208,17 @@ ESCOTT | http://gitlab.lcqb.upmc.fr/tekpinar/PRESCOTT
 VenusREM | https://github.com/tyang816/VenusREM
 RSALOR | https://github.com/3BioCompBio/RSALOR
 S3F | https://github.com/DeepGraphLearning/S3F
+SiteRM | https://github.com/songlab-cal/CherryML
 
-We would like to thank the GEMME team for providing model scores on an earlier version of the benchmark (ProteinGym v0.1), and the ProtSSN, SaProt, PoET, MULAN, VespaG, ProSST, ESCOTT, VenusREM, and RSALOR teams for integrating their model in the ProteinGym repo.
+We would like to thank the GEMME team for providing model scores on an earlier version of the benchmark (ProteinGym v0.1), and the ProtSSN, SaProt, PoET, MULAN, VespaG, ProSST, ESCOTT, VenusREM, RSALOR, and SiteRM teams for integrating their model in the ProteinGym repo.
 
 Special thanks the teams of experimentalists who developed and performed the assays that ProteinGym is built on. If you are using ProteinGym in your work, please consider citing the corresponding papers. To facilitate this, we have prepared a file (assays.bib) containing the bibtex entries for all these papers.
 
 ## Releases
 
 1. [ProteinGym_v1.0](https://zenodo.org/records/13932633): Initial release.
 2. [ProteinGym_v1.1](https://zenodo.org/records/13936340): Updates to reference file, and addition of ProtSSN and SaProt baselines.
+3. [ProteinGym_v1.2](https://zenodo.org/records/14997691): Added 8 baselines to the zero-shot DMS substitutions benchmark (eg., VenusREM, S3F, Escott). Added all mutation-level predictions for all baselines in supervised benchmarks.
 
 ## License
 This project is available under the MIT license found in the LICENSE file in this GitHub repository.
@@ -242,5 +243,5 @@ If you use ProteinGym in your work, please cite the following paper:
 - Website: https://www.proteingym.org/
 - NeurIPS proceedings: [link to abstract](https://papers.nips.cc/paper_files/paper/2023/hash/cac723e5ff29f65e3fcbb0739ae91bee-Abstract-Datasets_and_Benchmarks.html)
 - Preprint: [link to abstract](https://www.biorxiv.org/content/10.1101/2023.12.07.570727v1)
-- Zenodo: [link to zenodo](https://zenodo.org/records/13936340)
+- Zenodo: [link to zenodo](https://zenodo.org/records/14997691)
 - Pypi: [link to pypi](https://pypi.org/project/proteingym/)

benchmarks/DMS_supervised/indels/MSE/DMS_indels_MSE_DMS_level.csv

@@ -1,67 +1,67 @@
 DMS_id,ESM-1v Embeddings,MSA Transformer Embeddings,Tranception Embeddings
-B1LPA6_ECOSM_Russ_2020_indels,0.547,0.569,0.793
-BLAT_ECOLX_Gonzalez_2019_indels,0.366,0.313,0.273
-CAPSD_AAV2S_Sinai_2021_designed_indels,0.216,0.205,0.25
-CAPSD_AAV2S_Sinai_2021_library_indels,0.434,0.414,0.299
-HIS7_YEAST_Pokusaeva_2019_indels,0.161,0.198,0.158
-PTEN_HUMAN_Mighell_2018_indels,0.274,0.467,0.365
-P53_HUMAN_Kotler_2018_indels,0.315,0.383,0.348
-KCNJ2_MOUSE_Macdonald_2022_indels,0.517,0.671,0.391
-Q8EG35_SHEON_Campbell_2022_indels,0.57,0.624,0.512
-A4_HUMAN_Seuma_2022_indels,0.262,0.36,0.249
-S22A1_HUMAN_Yee_2023_abundance_indels,0.42,0.613,0.44
-S22A1_HUMAN_Yee_2023_activity_indels,0.427,0.606,0.397
-AMFR_HUMAN_Rocklin_2023_4G3O_indels,0.267,0.467,0.848
-ARGR_ECOLI_Rocklin_2023_1AOY_indels,0.138,0.259,0.204
-BBC1_YEAST_Rocklin_2023_1TG0_indels,0.224,0.328,0.391
-BCHB_CHLTE_Rocklin_2023_2KRU_indels,0.194,0.503,0.449
-CATR_CHLRE_Rocklin_2023_2AMI_indels,0.192,0.362,0.247
-CBPA2_HUMAN_Rocklin_2023_1O6X_indels,0.131,0.292,0.346
-CBX4_HUMAN_Rocklin_2023_2K28_indels,0.246,0.423,0.409
-CSN4_MOUSE_Rocklin_2023_1UFM_indels,0.087,0.292,0.288
-CUE1_YEAST_Rocklin_2023_2MYX_indels,0.216,0.361,0.56
-DN7A_SACS2_Rocklin_2023_1JIC_indels,0.282,0.547,0.549
-DNJA1_HUMAN_Rocklin_2023_2LO1_indels,0.128,0.25,0.162
-DOCK1_MOUSE_Rocklin_2023_2M0Y_indels,0.281,0.491,0.353
-EPHB2_HUMAN_Rocklin_2023_1F0M_indels,0.125,0.299,0.307
-FECA_ECOLI_Rocklin_2023_2D1U_indels,0.488,0.508,0.592
-HCP_LAMBD_Rocklin_2023_2L6Q_indels,0.091,0.267,0.238
-HECD1_HUMAN_Rocklin_2023_3DKM_indels,0.63,0.568,0.645
-ILF3_HUMAN_Rocklin_2023_2L33_indels,0.135,0.304,0.245
-MAFG_MOUSE_Rocklin_2023_1K1V_indels,0.257,0.548,0.493
-MBD11_ARATH_Rocklin_2023_6ACV_indels,0.272,0.377,0.597
-MYO3_YEAST_Rocklin_2023_2BTT_indels,0.392,0.612,0.514
-NKX31_HUMAN_Rocklin_2023_2L9R_indels,0.153,0.367,0.156
-NUSA_ECOLI_Rocklin_2023_1WCL_indels,0.154,0.284,0.527
-NUSG_MYCTU_Rocklin_2023_2MI6_indels,0.22,0.388,0.327
-OBSCN_HUMAN_Rocklin_2023_1V1C_indels,0.178,0.282,0.342
-ODP2_GEOSE_Rocklin_2023_1W4G_indels,0.46,0.682,0.543
-OTU7A_HUMAN_Rocklin_2023_2L2D_indels,0.216,0.422,0.56
-PIN1_HUMAN_Rocklin_2023_1I6C_indels,0.113,0.389,0.269
-PITX2_HUMAN_Rocklin_2023_2L7M_indels,0.495,0.773,0.605
-PKN1_HUMAN_Rocklin_2023_1URF_indels,0.082,0.178,0.156
-POLG_PESV_Rocklin_2023_2MXD_indels,0.198,0.495,0.384
-PR40A_HUMAN_Rocklin_2023_1UZC_indels,0.168,0.312,0.279
-PSAE_SYNP2_Rocklin_2023_1PSE_indels,0.24,0.259,0.311
-RAD_ANTMA_Rocklin_2023_2CJJ_indels,0.193,0.33,0.294
-RCD1_ARATH_Rocklin_2023_5OAO_indels,0.251,0.359,0.424
-RD23A_HUMAN_Rocklin_2023_1IFY_indels,0.154,0.349,0.29
-RPC1_BP434_Rocklin_2023_1R69_indels,0.484,0.511,0.431
-RS15_GEOSE_Rocklin_2023_1A32_indels,0.203,0.355,0.331
-SAV1_MOUSE_Rocklin_2023_2YSB_indels,0.531,0.708,0.648
-SDA_BACSU_Rocklin_2023_1PV0_indels,0.113,0.197,0.184
-SOX30_HUMAN_Rocklin_2023_7JJK_indels,0.195,0.336,0.566
-SPG2_STRSG_Rocklin_2023_5UBS_indels,0.16,0.256,0.262
-SPTN1_CHICK_Rocklin_2023_1TUD_indels,0.237,0.332,0.471
-SQSTM_MOUSE_Rocklin_2023_2RRU_indels,0.225,0.559,0.442
-SR43C_ARATH_Rocklin_2023_2N88_indels,0.223,0.323,0.235
-SRBS1_HUMAN_Rocklin_2023_2O2W_indels,0.153,0.366,0.201
-TCRG1_MOUSE_Rocklin_2023_1E0L_indels,0.118,0.22,0.329
-THO1_YEAST_Rocklin_2023_2WQG_indels,0.233,0.421,0.623
-TNKS2_HUMAN_Rocklin_2023_5JRT_indels,0.22,0.405,0.477
-UBE4B_HUMAN_Rocklin_2023_3L1X_indels,0.348,0.431,0.533
-UBR5_HUMAN_Rocklin_2023_1I2T_indels,0.196,0.344,0.288
-VG08_BPP22_Rocklin_2023_2GP8_indels,0.187,0.291,0.394
-VILI_CHICK_Rocklin_2023_1YU5_indels,0.133,0.215,0.184
-VRPI_BPT7_Rocklin_2023_2WNM_indels,0.235,0.496,0.474
-YNZC_BACSU_Rocklin_2023_2JVD_indels,0.1,0.343,0.303
+A4_HUMAN_Seuma_2022_indels,0.283,0.302,0.248
+AMFR_HUMAN_Tsuboyama_2023_4G3O_indels,0.323,0.345,0.713
+ARGR_ECOLI_Tsuboyama_2023_1AOY_indels,0.148,0.262,0.18
+B1LPA6_ECOSM_Russ_2020_indels,0.522,0.552,0.762
+BBC1_YEAST_Tsuboyama_2023_1TG0_indels,0.225,0.317,0.357
+BCHB_CHLTE_Tsuboyama_2023_2KRU_indels,0.224,0.361,0.319
+BLAT_ECOLX_Gonzalez_2019_indels,0.485,0.302,0.288
+CAPSD_AAV2S_Sinai_2021_designed_indels,0.441,0.205,0.217
+CAPSD_AAV2S_Sinai_2021_library_indels,0.614,0.362,0.321
+CATR_CHLRE_Tsuboyama_2023_2AMI_indels,0.163,0.303,0.233
+CBPA2_HUMAN_Tsuboyama_2023_1O6X_indels,0.157,0.217,0.302
+CBX4_HUMAN_Tsuboyama_2023_2K28_indels,0.26,0.413,0.397
+CSN4_MOUSE_Tsuboyama_2023_1UFM_indels,0.088,0.205,0.283
+CUE1_YEAST_Tsuboyama_2023_2MYX_indels,0.248,0.258,0.432
+DN7A_SACS2_Tsuboyama_2023_1JIC_indels,0.32,0.301,0.437
+DNJA1_HUMAN_Tsuboyama_2023_2LO1_indels,0.108,0.192,0.118
+DOCK1_MOUSE_Tsuboyama_2023_2M0Y_indels,0.357,0.335,0.341
+EPHB2_HUMAN_Tsuboyama_2023_1F0M_indels,0.114,0.252,0.246
+FECA_ECOLI_Tsuboyama_2023_2D1U_indels,0.471,0.449,0.598
+HCP_LAMBD_Tsuboyama_2023_2L6Q_indels,0.1,0.227,0.142
+HECD1_HUMAN_Tsuboyama_2023_3DKM_indels,0.747,0.564,0.656
+HIS7_YEAST_Pokusaeva_2019_indels,0.182,0.157,0.164
+ILF3_HUMAN_Tsuboyama_2023_2L33_indels,0.157,0.22,0.241
+KCNJ2_MOUSE_Macdonald_2022_indels,0.596,0.458,0.423
+MAFG_MOUSE_Tsuboyama_2023_1K1V_indels,0.311,0.312,0.262
+MBD11_ARATH_Tsuboyama_2023_6ACV_indels,0.37,0.363,0.458
+MYO3_YEAST_Tsuboyama_2023_2BTT_indels,0.383,0.512,0.495
+NKX31_HUMAN_Tsuboyama_2023_2L9R_indels,0.163,0.252,0.15
+NUSA_ECOLI_Tsuboyama_2023_1WCL_indels,0.179,0.26,0.472
+NUSG_MYCTU_Tsuboyama_2023_2MI6_indels,0.197,0.308,0.258
+OBSCN_HUMAN_Tsuboyama_2023_1V1C_indels,0.194,0.244,0.34
+ODP2_GEOSE_Tsuboyama_2023_1W4G_indels,0.516,0.534,0.53
+OTU7A_HUMAN_Tsuboyama_2023_2L2D_indels,0.209,0.414,0.361
+P53_HUMAN_Kotler_2018_indels,0.406,0.288,0.325
+PIN1_HUMAN_Tsuboyama_2023_1I6C_indels,0.12,0.197,0.224
+PITX2_HUMAN_Tsuboyama_2023_2L7M_indels,0.387,0.538,0.465
+PKN1_HUMAN_Tsuboyama_2023_1URF_indels,0.089,0.181,0.129
+POLG_PESV_Tsuboyama_2023_2MXD_indels,0.178,0.3,0.284
+PR40A_HUMAN_Tsuboyama_2023_1UZC_indels,0.221,0.268,0.243
+PSAE_PICP2_Tsuboyama_2023_1PSE_indels,0.219,0.234,0.313
+PTEN_HUMAN_Mighell_2018_indels,0.299,0.264,0.384
+Q8EG35_SHEON_Campbell_2022_indels,0.495,0.539,0.534
+RAD_ANTMA_Tsuboyama_2023_2CJJ_indels,0.19,0.263,0.224
+RCD1_ARATH_Tsuboyama_2023_5OAO_indels,0.306,0.265,0.34
+RD23A_HUMAN_Tsuboyama_2023_1IFY_indels,0.151,0.186,0.222
+RPC1_BP434_Tsuboyama_2023_1R69_indels,0.476,0.462,0.398
+RS15_GEOSE_Tsuboyama_2023_1A32_indels,0.212,0.335,0.236
+S22A1_HUMAN_Yee_2023_abundance_indels,0.472,0.471,0.44
+S22A1_HUMAN_Yee_2023_activity_indels,0.487,0.373,0.397
+SAV1_MOUSE_Tsuboyama_2023_2YSB_indels,0.56,0.625,0.569
+SDA_BACSU_Tsuboyama_2023_1PV0_indels,0.157,0.163,0.132
+SOX30_HUMAN_Tsuboyama_2023_7JJK_indels,0.188,0.351,0.383
+SPG2_STRSG_Tsuboyama_2023_5UBS_indels,0.161,0.255,0.229
+SPTN1_CHICK_Tsuboyama_2023_1TUD_indels,0.245,0.262,0.299
+SQSTM_MOUSE_Tsuboyama_2023_2RRU_indels,0.232,0.369,0.396
+SR43C_ARATH_Tsuboyama_2023_2N88_indels,0.219,0.25,0.271
+SRBS1_HUMAN_Tsuboyama_2023_2O2W_indels,0.161,0.288,0.194
+TCRG1_MOUSE_Tsuboyama_2023_1E0L_indels,0.109,0.174,0.216
+THO1_YEAST_Tsuboyama_2023_2WQG_indels,0.205,0.247,0.423
+TNKS2_HUMAN_Tsuboyama_2023_5JRT_indels,0.198,0.349,0.377
+UBE4B_HUMAN_Tsuboyama_2023_3L1X_indels,0.362,0.397,0.463
+UBR5_HUMAN_Tsuboyama_2023_1I2T_indels,0.236,0.302,0.258
+VG08_BPP22_Tsuboyama_2023_2GP8_indels,0.177,0.261,0.266
+VILI_CHICK_Tsuboyama_2023_1YU5_indels,0.173,0.177,0.142
+VRPI_BPT7_Tsuboyama_2023_2WNM_indels,0.289,0.449,0.393
+YNZC_BACSU_Tsuboyama_2023_2JVD_indels,0.108,0.216,0.189