Add initial examples (#1)

Co-authored-by: ElliottKasoar <ElliottKasoar@users.noreply.github.com>
Co-authored-by: joehart2001 <jh2536@cam.ac.uk>

Author: ElliottKasoar

.dockerignore

@@ -0,0 +1 @@
+.venv

.gitignore

@@ -1,5 +1,8 @@
+*.json
+*.html
 *.dat
 *.lock
+!*dvc.lock
 *.log
 *.pyc
 *.swo
@@ -10,6 +13,7 @@
 *.extxyz
 *.yml
 *.yaml
+!dvc.yaml
 *.pdf
 *.hdf5
 *.traj

CITATION.cff

@@ -7,7 +7,7 @@ authors:
     given-names: Elliott
     orcid: "https://orcid.org/0009-0005-2015-9478"
   - family-names: Hart
-    given-names: Joe
+    given-names: Joseph
 identifiers:
   - type: doi
     value: 10.5281/zenodo.16904444

Dockerfile

@@ -0,0 +1,31 @@
+# Install uv
+FROM python:3.12-slim
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
+
+# Install git, gcc etc.
+RUN apt-get update && apt-get upgrade -y && apt-get install -y git build-essential
+
+# Change the working directory to the `app` directory
+WORKDIR /app
+
+# Enable bytecode compilation
+ENV UV_COMPILE_BYTECODE=1
+
+# Copy from the cache instead of linking since it's a mounted volume
+ENV UV_LINK_MODE=copy
+
+# Install dependencies
+RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=bind,source=uv.lock,target=uv.lock \
+    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
+    uv sync -v --locked --no-install-project
+
+# Copy contents into the image
+ADD . /app
+
+# Sync the project
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --locked
+
+# Run app.py
+CMD ["uv", "run", "run_app.py"]

compose.yaml

@@ -0,0 +1,8 @@
+services:
+  app:
+    image: benchmark-app
+    ports:
+      - 8050:8050
+    working_dir: /app
+    volumes:
+      - ./:/app

conftest.py

@@ -0,0 +1,35 @@
+"""
+Configure pytest.
+
+Based on https://docs.pytest.org/en/latest/example/simple.html.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+
+def pytest_addoption(parser):
+    """Add flag to run tests for extra MLIPs."""
+    parser.addoption(
+        "--run-slow",
+        action="store_true",
+        default=False,
+        help="Run slow benchmarks",
+    )
+
+
+def pytest_configure(config):
+    """Configure pytest to include marker for extra MLIPs."""
+    config.addinivalue_line("markers", "slow: mark test as slow calculations")
+
+
+def pytest_collection_modifyitems(config, items):
+    """Skip tests if marker applied to unit tests."""
+    if config.getoption("--run-slow"):
+        # --run-slow given in cli: do not skip tests
+        return
+    skip_slow = pytest.mark.skip(reason="need --run-slow option to run")
+    for item in items:
+        if "slow" in item.keywords:
+            item.add_marker(skip_slow)

docs/source/developer_guide/add_benchmarks.rst

@@ -44,18 +44,18 @@ Running Calculations
 2. Write a script that will run the MLIP calculations of interest for each model being tested.
 
 The file should be named ``calc_[benchmark_name].py``,
-and placed in ``mlip_testing/calcs/[benchmark_name]``.
+and placed in ``mlip_testing/calcs/[category]/[benchmark_name]``.
 
 While not a requirement, we recommend placing input files in
-``mlip_testing/calcs/[benchmark_name]/data``, and output files in
-``mlip_testing/calcs/[benchmark_name]/outputs``, for consistency.
+``mlip_testing/calcs/[category]/[benchmark_name]/data``, and output files in
+``mlip_testing/calcs/[category]/[benchmark_name]/outputs``, for consistency.
 
 The test contained in this file may be runnable as a standalone script,
 but it should also be possible to run with ``pytest``, e.g.:
 
 .. code-block:: bash
 
-    pytest -v -s mlip_testing/calcs/[benchmark_name]/calc_[benchmark_name].py
+    pytest -v -s mlip_testing/calcs/[category]/[benchmark_name]/calc_[benchmark_name].py
 
 
 .. note::
@@ -121,7 +121,7 @@ b. Defining a ``ZnTrack`` node to run via ``mlipx``:
 
 The process of running these is largely as
 `described by mlipx <https://mlipx.readthedocs.io/en/latest/quickstart/cli.html>`_,
-including running ``dvc init`` in ``mlip_testing/calcs/[benchmark_name]``.
+including running ``dvc init`` in ``mlip_testing/calcs/[category]/[benchmark_name]``.
 
 .. note::
 
@@ -229,12 +229,13 @@ directory to be accessed by the app.
 As with the script created in :ref:`calculations`, we create a new file to be run by ``pytest``,
 containing a function beginning with ``test_`` to launch the analysis.
 
-In this case, we name the file ``mlip_testing/analysis/[benchmark_name]/analyse_[benchmark_name].py``,
+In this case, we name the file
+``mlip_testing/analysis/[category]/[benchmark_name]/analyse_[benchmark_name].py``,
 such that it can be run using:
 
 .. code-block:: bash
 
-    pytest -v -s mlip_testing/analysis/[benchmark_name]/analyse_[benchmark_name].py
+    pytest -v -s mlip_testing/analysis/[category]/[benchmark_name]/analyse_[benchmark_name].py
 
 
 In order to automatically generate the components for our application, we will make use
@@ -256,7 +257,7 @@ For ``@build_table``, the value returned should be of the form:
 This will generate a table with columns for each metric, as well as "MLIP", "Score",
 and "Rank" columns. Tooltips for each column header can also be set by the decorator,
 as well as the location to save the JSON file to be loaded when building the app,
-which typically would be placed in ``mlip_testing/app/data/[benchmark_name]``.
+which typically would be placed in ``mlip_testing/app/data/[category]/[benchmark_name]``.
 
 Every benchmark should have at least one of these tables, which includes
 the score for each metric, and allowing the table to calculate an overall score for the
@@ -308,8 +309,11 @@ which allows the value returned by a function to be used directly as a parameter
 for other functions.
 
 If your benchmark contains structures to be visualised, or images to be loaded, these
-should be saved to ``mlip_testing/app/data/[benchmark_name]``, as they must be added as
-``assets`` to be loaded into the app.
+should be saved to ``mlip_testing/app/data/[category]/[benchmark_name]``, as they must
+be added as ``assets`` to be loaded into the app.
+
+Absolute paths to ``mlip_testing/app`` and ``mlip_testing/calcs`` can be imported for
+convenience.
 
 .. note::
 
@@ -320,10 +324,12 @@ should be saved to ``mlip_testing/app/data/[benchmark_name]``, as they must be a
 
     from mlip_testing.analysis.utils.decorators import build_table, plot_parity
     from mlip_testing.analysis.utils.utils import mae
+    from mlip_testing.app import APP_ROOT
+    from mlip_testing.calcs import CALCS_ROOT
     from mlip_testing.calcs.models.models import MODELS
 
-    CALC_PATH = Path(__file__).parent.parent.parent / "calcs" / [benchmark_name] / "outputs"
-    OUT_PATH = Path(__file__).parent.parent.parent / "app" / "data" / [benchmark_name]
+    CALC_PATH = CALCS_ROOT / [category] / [benchmark_name] / "outputs"
+    OUT_PATH = APP_ROOT / "data" / [category] / [benchmark_name]
 
     REF_VALUES = {"path_b": 0.27, "path_c": 2.5}
 

docs/source/user_guide/benchmarks/index.rst

@@ -0,0 +1,9 @@
+==========
+Benchmarks
+==========
+
+.. toctree::
+    :maxdepth: 3
+
+    surfaces
+    nebs

docs/source/user_guide/benchmarks/nebs.rst

@@ -0,0 +1,48 @@
+====
+NEBs
+====
+
+Li diffusion
+============
+
+Summary
+-------
+
+Performance in predicting activation energies of Li diffusion along the [010] and [001]
+directions of LiFePO_4.
+
+Metrics
+-------
+
+1. [010] (path B) energy barrier error
+
+The initial and final structures for the diffusion of lithium along [010] are created
+through deletion an atom from the initial structure. These structures are relaxed,
+and the Nudged Elastic Band method is used to calculate the energy barrier. This is
+compared to the reference activation energy for this path.
+
+
+2. [001] (path C) energy barrier error
+
+The initial and final structures for the diffusion of lithium along [001] are created
+through deletion an atom from the initial structure. These structures are relaxed,
+and the Nudged Elastic Band method is used to calculate the energy barrier. This is
+compared to the reference activation energy for this path.
+
+Computational cost
+------------------
+
+Medium: tests are likely to take several minutes to run on CPU.
+
+
+Data availability
+-----------------
+
+Input structure:
+
+* Downloaded from Materials Project (mp-19017): https://doi.org/10.17188/1193803
+
+Reference data:
+
+* Manually taken from https://doi.org/10.1149/1.1633511.
+* Meta-GGA (Perdew-Wang) exchange correlation functional

docs/source/user_guide/benchmarks/surfaces.rst

@@ -0,0 +1,103 @@
+========
+Surfaces
+========
+
+OC157
+=====
+
+Summary
+-------
+
+Performance in predicting relative energies between three configurations for 157
+molecule-surface combinations.
+
+Metrics
+-------
+
+1. Energy error
+
+How accurate all relatve energy predictions are.
+
+For each group of three structures, the relative energies are calculated for all pairs
+of structures. Models receive a score based on the mean difference between these
+predictions and the reference, averaged over all pairs and over all combinations.
+
+2. Ranking error
+
+Whether the most and least stable strucutres are predicted.
+
+For each group of three structures, the relative energies are calculated for all pairs
+of structures. Models receive a score of 0, 0.5, or 1, based on whether the predicted
+lowest and highest energy pairs match the reference predictions, and this is averaged
+for all 157 combinations.
+
+Computational cost
+------------------
+
+Low: tests are likely to take a couple of minutes to run on CPU.
+
+Data availability
+-----------------
+
+Input data:
+
+* Surfaces were taken from the Open Catalyst Challenge 2023
+
+  * L. Chanussot, A. Das, S. Goyal, T. Lavril, M. Shuaibi, M. Riviere, K. Tran, J. Heras-Domingo, C. Ho, W. Hu, A. Palizhati, A. Sriram, B. Wood, J. Yoon, D. Parikh, C. L. Zitnick, and Z. Ulissi, “Open Catalyst 2020 (OC20) dataset and community challenges,” ACS Catal., vol. 11, pp. 6059–6072, May 2021.
+  * R. Tran, J. Lan, M. Shuaibi, B. M. Wood, S. Goyal, A. Das, J. Heras-Domingo, A. Kolluru, A. Rizvi, N. Shoghi, A. Sriram, F. Therrien, J. Abed, O. Voznyy, E. H. Sargent, Z. Ulissi, and C. L. Zitnick, “The Open Catalyst 2022 (OC22) data set and challenges for oxide electro catalysts,” ACS Catal., vol.13, pp. 3066–3084, Mar. 2023.
+
+* Structures containing oxygen (O) and several transition metals (Co, Cr, Fe, Mn, Mo,
+  Ni, V and W) were exlcuded due to Hubbard U correction
+
+Reference data:
+
+* Same as input data
+* PBE-D3(BJ), MPRelaxSet settings
+
+
+S24
+===
+
+Summary
+-------
+
+Performance in predicting adsorption energies for a diverse set of surfaces and adsorbates.
+
+Metrics
+-------
+
+Adsorption energy error
+
+For each combination of surface, molecule, and surface + molecule, the adsorption
+energy is calculated by taking the difference between the energy of the surface +
+molecule and the sum of individual surface and molecule energies. This is compared to
+the reference adsorption energy, calculated in the same way.
+
+Computational cost
+------------------
+
+Very low: tests are likely to take less than a minute to run on CPU.
+
+Data availability
+-----------------
+
+Input data:
+
+* Structures were taken from an amalgamation of published and unpublished works, including:
+
+  * Y. S. Al-Hamdani, M. Rossi, D. Alfè, T. Tsatsoulis, B. Ramberger, J. G. Brandenburg, A. Zen, G. Kresse, A. Grüneis, A. Tkatchenko, and A. Michaelides, “Properties of the water to boron nitride interaction: From zero to two dimensions with benchmark accuracy,” J. Chem. Phys., vol. 147, p. 044710, July 2017. 35
+  * J. G. Brandenburg, A. Zen, M. Fitzner, B. Ramberger, G. Kresse, T. Tsatsoulis, A. Grüneis, A. Michaelides, and D. Alfè, “Physisorption of water on graphene: Subchemical accuracy from many- body electronic structure methods,” J. Phys. Chem. Lett., vol. 10, pp. 358–368, Feb. 2019.
+  * C. Ehlert, A. Piras, and G. Gryn’ova, “CO2 on graphene: Benchmarking computational approaches to noncovalent interactions,” ACS Omega, vol. 8, pp. 35768–35778, Oct. 2023.
+  * T. Tsatsoulis, S. Sakong, A. Groß, and A. Grüneis, “Reaction energetics of hydrogen on Si(100) surface: A periodic many-electron theory study,” J. Chem. Phys., vol. 149, p. 244105, Dec. 2018.
+  * T. Tsatsoulis, F. Hummel, D. Usvyat, M. Schütz, G. H. Booth, S. S. Binnie, M. J. Gillan, D. Alfè, A. Michaelides, and A. Grüneis, “A comparison between quantum chemistry and quantum Monte Carlo techniques for the adsorption of water on the (001) LiH surface,” J. Chem. Phys., vol. 146, p. 204108, May 2017.
+  *  H.-Z. Ye and T. C. Berkelbach, “Ab initio surface chemistry with chemical accuracy,” arXiv preprint arXiv:2309.14640, 2023.
+  * P. G. Lustemberg, P. N. Plessow, Y. Wang, C. Yang, A. Nefedov, F. Studt, C. Wöll, and M. V. Ganduglia-Pirovano, “Vibrational frequencies of cerium-oxide-bound CO: A challenge for conventional dft methods,” Phys. Rev. Lett., vol. 125, p. 256101, Dec. 2020.
+  * B. X. Shi, A. Zen, V. Kapil, P. R. Nagy, A. Grüneis, and A. Michaelides, “Many-body methods for surface chemistry come of age: Achieving consensus with experiments,” J. Am. Chem. Soc., vol. 145, pp. 25372–25381, Nov. 2023.
+  *  N. Hanikel, X. Pei, S. Chheda, H. Lyu, W. Jeong, J. Sauer, L. Gagliardi, and O. M. Yaghi, “Evolution of water structures in metal-organic frameworks for improved atmospheric water harvesting,” Science, vol. 374, pp. 454–459, 2021.
+  * F. Berger, M. Rybicki, and J. Sauer, “Molecular dynamics with chemical accuracy–Alkane adsorption in acidic zeolites,” ACS Catal., vol. 13, pp. 2011–2024, 2023.
+  * F. Berger and J. Sauer, “Dimerization of linear butenes and pentenes in an acidic zeolite (H-MFI),” Angew. Chem., Int. Ed., vol. 60, pp. 3529–3533, 2021.
+
+Reference data:
+
+* Same as input data
+* PBE-D3(BJ), MPRelaxSet settings