Performance validation of the trained M3GNET model

[Juo-kim]

Hi,

I trained the m3gnet model on MPtrj Dataset, which is used for training chgnet.
(https://figshare.com/articles/dataset/Materials_Project_Trjectory_MPtrj_Dataset/23713842.)

Good performance was obtained during the training process, and the performance was decent compared to the pretrained model you provided.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ test_Energy_MAE │ 0.1104562059044838 │
│ test_Energy_RMSE │ 0.13586871325969696 │
│ test_Force_MAE │ 0.09391890466213226 │
│ test_Force_RMSE │ 0.15049749612808228 │
│ test_Site_Wise_MAE │ 0.0 │
│ test_Site_Wise_RMSE │ 0.0 │
│ test_Stress_MAE │ 0.0 │
│ test_Stress_RMSE │ 0.0 │
│ test_Total_Loss │ 0.04630613327026367 │
└───────────────────────────┴───────────────────────────┘

However, when I ㅣloadedd the trained model again for verification and predicted the potential energy of the structure, the performance was completely different. Can you tell me why this problem is?

[shyuep]

You will need to show your save and loading code. The likely reason is that you did not save and load the model properly as a Potential class.

[Juo-kim]

Thank you for replying,

First of all, for the test, the mptrj dataset was divided into several parts and the training was conducted on those data. When conducting the training, it was conducted after converting the data into a graph and referring to the sample code. The code I used is as follows.

dataset = MGLDataset(
    filename = './partial_10/dgl_graph.bin',
    filename_lattice = './partial_10/lattice.pt',
    filename_state_attr = './partial_10/state_attr.pt',
    filename_labels = './partial_10/labels.json',
    threebody_cutoff=4.0
    #structures=structures,
    #converter=converter,
    #labels=labels,
    #clear_processed = True
    )

train_data, val_data, test_data = split_dataset(
    dataset,
    frac_list=[0.9, 0.05, 0.05],
    shuffle=True,
    random_state=42,
)

train_loader, val_loader, test_loader = MGLDataLoader(
    train_data=train_data,
    val_data=val_data,
    test_data=test_data,
    collate_fn=collate_fn_efs,
    batch_size=16,
    num_workers= 0,
)

model = M3GNet(
    element_types=element_types,
    is_intensive=False,
)
lit_module = PotentialLightningModule(model=model)

# If you wish to disable GPU or MPS (M1 mac) training, use the accelerator="cpu" kwarg.
logger = CSVLogger("logs", name="M3GNet_training")
# Inference mode = False is required for calculating forces, stress in test mode and prediction mode
trainer = pl.Trainer(max_epochs=50, accelerator="cuda", logger=logger, inference_mode=False)
trainer.fit(model=lit_module, train_dataloaders=train_loader, val_dataloaders=val_loader)

# test the model, remember to set inference_mode=False in trainer (see above)
trainer.test(dataloaders=test_loader)

# save trained model
model_export_path = "./trained_model/"
model.save(model_export_path)

# load trained model

and when load the dataset I loaded the model by using the load_model, and wrapped with Potential . Then, I calculated the potential energies, but it those not match well with the DFT energies.

trained = matgl.load_model('./trained_model/')
pm = Potential(trained)
calc_matgl = M3GNetCalculator(pm)

ase_adaptor = AseAtomsAdaptor()
atoms = ase_adaptor.get_atoms(structure)
atoms.calc = calc_matgl
potential_energy = atoms.get_potential_energy()        
predicted_energy = potential_energy / len(structure)

The hyperparameter of the m3gnet model is the same as the pretrained model ("M3Get-MP-2021.2.8-PES"), except for the model I made where element_ref is a null value. Did I do something wrong or miss something during the model's training process?

[donghwa33]

Recently, I had a similar problem to yours. I downgraded the matgl to 0.7.1, torch to 2.0.1 and clear the cache of dgl_graph and it worked for me. In my case, I think the problem was the cached dgl_graph. Hope this helps.