Scope question: would a materials spectroscopy benchmark task fit Matbench Discovery?

[nkwork9999]

Hi Matbench Discovery maintainers,

I would like to ask a scope question before preparing any PR or prototype:
would a materials spectroscopy benchmark task be considered in scope for
Matbench Discovery?

The rough idea is to explore whether an additional experimental-spectroscopy
task family could complement the current discovery / geo-opt / phonons /
diatomics tasks. Example directions could include:

• NIR spectra -> material, process, or property prediction
• NMR spectra -> molecular, functional-group, or material-chemistry prediction
• possible materials focus: polymers, emulsions, additives, fillers, and
  semiconductor packaging materials

I understand that Matbench Discovery currently focuses on inorganic crystal
discovery, geometry optimization, phonons, thermal conductivity, and interatomic
potentials. So experimental spectroscopy may simply be outside the intended
scope of this repository.

Before going further, I wanted to ask:

1. Would a materials spectroscopy task family be welcome here if it came with
   open data, fixed splits, metrics, reproducible baselines, and a clear
   prediction-file format?
2. If this is outside the intended scope of Matbench Discovery, would you
   recommend building it as a separate Matbench-inspired benchmark instead?
3. For models submitted to the existing Matbench Discovery tasks, would it be
   acceptable to mention NIR/NMR-related external training or validation data in
   the model card's notes field, without treating it as a scored task?

If this sounds potentially in scope, I can prepare a more concrete proposal with:

• candidate datasets and licenses
• train/validation/test split policy
• baseline models
• metrics
• prediction-file format
• minimal website/table integration plan

Thanks for maintaining Matbench Discovery.

[janosh]

@nkwork9999 thanks for the suggestion! spectroscopy could be a very interesting next direction since we just added finite-temperature molecular dynamics as a new benchmark task in #344. seems like a natural extension of MLIP benchmarking to incentivize the field to make them more experimentally relevant. we've been a bit simulation siloed so far. i imagine the current models won't be good initially so the point you raise about good baselines to compare against is important, to give readers a feeling how far away from usefulness we are.

would it be acceptable to mention NIR/NMR-related external training or validation data in the model card's notes field

if people start training on NIR/NMR datasets, we'll just add those datasets to datasets.yml

matbench-discovery/data/datasets.yml

Lines 1 to 13 in 2b0d3ce

	WBM:
	name: WBM Benchmark Test Set
	url: https://github.com/janosh/matbench-discovery/blob/main/data/wbm/readme.md
	doi: https://doi.org/10.1038/s41524-020-00481-6
	n_structures: 256963 # 257_487 before filtering suspicious data, see data/wbm/compile_wbm_test_set.py
	n_materials: 215488 # based on number of unique Aflow-style structure prototypes
	n_stable_materials: 42825
	n_uniq_stable_materials: 32942
	open: true
	static: true
	optimade_api: null
	native_api: null
	date_created: 2021-01-01

and then refer to them as additional training sets in each model's metadata

happy to discuss further details here or in a PR if you already located some reference data