AutoML checkpointing?

[ZviBaratz]

Hi,

When using ray, I am getting the following message:

2022-06-23 16:24:11,299	WARNING function_runner.py:603 -- Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.

I was wondering whether "checkpointing" functionality is available when using AutoML with ray, i.e. whether it is possible to store intermediate results in some work directory and run incrementally (e.g. at nights, when the computer is idle, etc).

Thank you very much for your help and all the great work!

[sonichi]

Currently, the way to do warm start is via the starting_points argument. Are you thinking of normal termination of an AutoML run followed by another AutoML run or a forced termination of an AutoML run followed by another AutoML run that recovers from the failure? For the former, warm-start + logging can work. For the latter, it also works when not using ray. But when using ray, the log is not written when the termination is forced and that needs an improvement.

[ZviBaratz]

Perfect, thank you! I am thinking of the normal termination case, I'm just not entirely clear on whether using starting_points will take into account previous trials (i.e. continue the search rather than restart it from the previous iteration's endpoint). Does pointing to an existing log_file_name do that?

[sonichi]

Using starting_points will not take into account previous trials. There is another way to take into account all previous trials in flaml.tune. flaml.AutoML currently doesn't utilize that feature yet, but it's not hard to add that feature: reconstruct the points_to_evaluate and evaluated_rewards from the log and pass it to the search algorithm.

[ZviBaratz]

That is what I was looking for.
I'll try to implement it locally and if it goes well maybe try to come up with a PR at some point. I think it could be a really cool feature to have.