Molnsutil state

[briandrawert]

No description provided.

[ahellander]

Not that it is so important, but we could change the name to "get_persisistent_storage_config"

[briandrawert]

Agreed

[ahellander]

Great improvements! I tested to "come back" to the DistribtedEnsemble after having the laptop closed.

Bug: It seems like that it is always using one less engine as the number of available engines when it calculates the chunk_size. I.e. I had 2 engines and got chunk_size=1. Setting the 'num_engines=1' as an argument causes an error.

Enhancement: This might be what you were mentioning before, but if 'storage_mode' is set to "Persistent", not only the results but also the ensemble state file should be written to S3/Swift, not to the controller fs. This of course introduces the complication that there will be a global name for ensembles, so we might need to add an API function to list the names of available ensembles and sweeps. Also, since the S3 bucket name is unique for a controller and private as default, sharing data between different users/controllers will require being able to set the S3/Swift bucket to public, right?

I also wonder if we should change the behavior and have the async behavior be the default for both ensembles and sweeps, and optionally use the progress bar but only after setting the flag "synchronous=True". Also, it would be very nice to mimic the behavior of IP parallels map_aynch, i.e. that we can iterate over trajectories as they arrive, i.e.:

ensemble = DistributedEnsemble(...)
res = ensemble.run(mapper=g)
for r in res.results:
print r

This increases the complexity, but will make it more useful for building more complex flows and still having the benefit of all the book-keeping in the DE, and PS classes.