Skip to content

Conversation

@scarlehoff
Copy link
Member

@scarlehoff scarlehoff commented Jan 9, 2026

With this PR we can now submit to multiple nodes in parallel (technically even to multiple clusters if you are feeling lucky).
The memory footprint of mongodb itself is much smaller as well.

Now an instance of n3fit can only spawn one single mongo worker, so it doesn't matter whether they are running in the same node or not, every worker connects to the database in the same manner. The num mongo workers option has been removed.
Now when a parallel hyperopt is going to run, each worker checks whether there's a database already running, if there is none, it starts the database and writes down the address. All other instances of n3fit will find an address and will try to connect.

There's also no longer a --restart option. Every new run is always a --restart. If you didn't want to continue a previous run just change the name of the runcard, but overwritting previous runs is very impolite.

Also, the database is no longer written in a separate folder then compressed but only when finishing cleanly and so on. Now the database is always in the nnfit folder, it will be compressed at vp-upload but not before (this is a problem because it is so big...), but only if the --upload-db option is used. Otherwise it skips the database during the upload. This makes the hyperopt not as heavy on the nnpdf server.

To do:

  • Update the documentation to reflect these changes
  • Make sure the automatic restart of mongodb works well. In particular finding automatically in which node the database is running. Seems to work but maybe a lockfile to ensure two databases don't start at once.
  • How to reduce the storage size of the database itself. I'm a bit baffled about this at the moment, I don't understand why the database is so damn huge. At least not upload it.
  • Remove the hash files in the working directory. Not sure whether this is possible but it is quite ugly. Can't find a way to do this 🤷‍♂️

Base automatically changed from update_deps_for_ruamel_019 to master January 11, 2026 12:00
n3fit run, server now is started only if a connection to a database
fails, --restart is removed, now only restart is allowed
@scarlehoff
Copy link
Member Author

This is now ready.

The only point I haven't been able to really fix is the creation of the cache hashes in the working directory. One solution is to run each worker in a separate node and only the one holding the database locally. But this is a cluster-dependent solution.

Btw, in principle you can run in parallel in different folders, nodes or computer, and the trials.json file will be always synchronized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants