Feat/add folmsbee conformer benchmark#429
Conversation
|
Hi @lwehrhan, thank you for your PR and its looking great overall! A few things:
Once we've got the data file uploaded, i think we can make a few changes to the calc script for consistency with similar benchmarks, but i think the changes will be minor. Just a note, make sure you to fetch any changes i've made before working locally, otherwise your next push may overwrite my changes. thanks! |
| # Add D3 calculator for this test | ||
| calc = model.add_d3_calculator(calc) | ||
|
|
||
| data_path = Path(__file__).parent / "data" / "folmsbee_dataset.json" |
There was a problem hiding this comment.
| data_path = Path(__file__).parent / "data" / "folmsbee_dataset.json" | |
| data_path = ( | |
| download_s3_data( | |
| filename="Folmsbee.zip", | |
| key="inputs/conformers/Folmsbee/Folmsbee.zip", | |
| ) | |
| / "Folmsbee" | |
| ) |
| import pytest | ||
| from tqdm import tqdm | ||
|
|
||
| from ml_peg.models.get_models import load_models |
There was a problem hiding this comment.
| from ml_peg.models.get_models import load_models | |
| from ml_peg.calcs.utils.utils import download_s3_data | |
| from ml_peg.models.get_models import load_models |
|
Hey @lwehrhan, we've just merged a PR so that you can tag your benchmark with the mlip-auditl, add your logo and have your own dedicated tab (PR #434). Please see the framework credit tags docs |
joehart2001
force-pushed
the
feat/add-folmsbee-conformer-benchmark
branch
from
e03bf30 to
efb1ea1
Compare
…b.com/lwehrhan/ml-peg into feat/add-folmsbee-conformer-benchmark
| Name of model and model object to get calculator. | ||
| """ | ||
| model_name, model = mlip | ||
| model.default_dtype = "float64" |
|
|
||
| (out_path / "model_output.json").write_text( | ||
| benchmark.model_output.model_dump_json() | ||
| ) |
| results[model_name] = mae( | ||
| conformer_energies["ref"], conformer_energies[model_name] | ||
| ) | ||
| return results |
There was a problem hiding this comment.
IIUC, here we are putting all conformers into a single flat list then taking the mae, whereas in the upstream benchmark we compute per-molecule MAEs which we then average to give avg_mae.
| i = int(conf_str) | ||
| molecule = result_by_name[mol_name] | ||
|
|
||
| results[model_name].append(float(molecule.predicted_energy_profile[i])) |
There was a problem hiding this comment.
This might be None if a molecule had an unsupported element.
|
|
||
| [tool.uv.sources] | ||
| asemolec = { git = "https://github.com/imagdau/aseMolec.git" } | ||
| mlipaudit = { git = "https://github.com/instadeepai/MLIPAudit.git", branch = "mlpeg-migration" } |
There was a problem hiding this comment.
remark: Note that this will have to be udpated to main at some point, either once everything is migrated or per-benchmark. Probably the former.
4 participants
Pre-review checklist for PR author
PR author must check the checkboxes below when creating the PR.
Summary
The Folmsbee dataset of low-energy conformers of drug-like molecules. The differences in energy are smaller compared to the Wiggle500 dataset and it features a greater number of molecules. The highest available level of theory for energy evaluations to be used as ground-truth is DLPNO-CCSD(T). This is a test for moving the benchmarks of mlip-audit into this repository. I have included an analysis script for this benchmark, however would like to kindly ask for assistance with building and harmonizing the Dash layout.
Linked issue
Resolves #427
Progress
Testing
New decorators/callbacks