An error in saving and restoring the model

[DiMY-CN]

Hi, I am grateful to the author for providing a very handy tool and communication platform from which I have benefited a lot. However, I am currently experiencing a problem with model saving and restoring. When I try to save a model, the following prompt appears:

INFO:tensorflow:C:/Users/oy135/Jupyter_code/PINN_train/model/model.ckpt-500.ckpt is not in all_model_checkpoint_paths. Manually adding it.INFO:tensorflow:C:/Users/oy135/Jupyter_code/PINN_train/model\model.ckpt-500.ckpt.data-00000-of-00001
INFO:tensorflow:1400
INFO:tensorflow:C:/Users/oy135/Jupyter_code/PINN_train/model\model.ckpt-500.ckpt.index
INFO:tensorflow:1400
INFO:tensorflow:C:/Users/oy135/Jupyter_code/PINN_train/model\model.ckpt-500.ckpt.meta
INFO:tensorflow:29200

Below is the code I used to save the model, this is just an attempt, I have done much larger model calculations before and the code works fine, but there is always a problem with saving and reading the model

model = dde.Model(data, net)
model.compile("adam", lr=0.001, metrics=["l2 relative error"], decay=("inverse time", 3000, 0.9))
checker = dde.callbacks.ModelCheckpoint(
    "C:/Users/oy135/Jupyter_code/PINN_train/model/model.ckpt", save_better_only=True, period=500
)
losshistory, train_state = model.train(iterations=1000, callbacks=[checker])

When I try to restore the model, I get the following error:

RuntimeError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint
NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint.

I've only excerpted a portion of the reported errors here, and combined with the hints when saving, it appears that the model.ckpt-500.ckpt file encountered a problem during saving. By looking at the save directory, I found that the file is indeed missing.

Below is the code I used to restore the model：

model = dde.Model(data, net)
model.compile("adam", lr=0.001, metrics=["l2 relative error"], decay=("inverse time", 3000, 0.9))
model.restore(save_path='C:/Users/oy135/Jupyter_code/PINN_train/model/model.ckpt-500.ckpt', verbose=1)

May I ask what is the reason for this?

[lululxvi]

It is unclear to me what the problem is. Are you able to save/restore the model using the demos in DeepXDE?

[DiMY-CN]

I've learned why the problem occurs, and the reason for the error is because the kernel has already trained a model. To restore the other model, the kernel needs to be restarted. Thanks for the answer.