SIGMA mining issue with windows machines >64 cores

[mjmacleod]

Describe the issue

Mining on a windows machine with more than 64 threads results in inability to use all of the threads, e.g. on an 88 thread machine only 44 will be properly utilised

Can you reliably reproduce the issue?

If so, please list the steps to reproduce below:

1. Install Gulden on a machine with more than 64 threads
2. Attempt to mine with as many threads as possible
3. CPUs will not be fully/properly utilised

Expected behaviour

UI should show the correct number of cores on the slider and all cores should be utilised.

Actual behaviour

Wrong maximum number of cores shown, and even if forced via RPC not all cores are utilised.

What version of Gulden are you using?

2.1.0.19, but presents in all current versions (SIGMA)

Machine specs:

• OS: Windows
• CPU: Dual core xeon machine

Any extra information that might be useful in the debugging process.

See https://bitsum.com/general/the-64-core-threshold-processor-groups-and-windows/ for an explanation of why this happens.

[mjmacleod]

Additional complications - our windows builds make use of mingw-w64 with pthreads.

mingw-w64 does not expose any of the windows thread group API stuff (or any functions of its own that wrap them) - nor does it expose pthreads 'sched_setaffinity' (its unclear if 'sched_setaffinity' would actually help anyway).

Porting off of pthreads would be a major change and likely will have lots of complications in our dependency libraries - the easiest solution here might be to try and get the mingw team to add processor group APIs to mingw.

[mjmacleod]

Also worth investigating - what happens if running the linux version in WSL - does WSL suffer from the same threading limitations or does that work correctly?

[mjmacleod]

A not ideal but reasonable workaround for this for now is to simply run two copies of the program, e.g.:
"start /NODE 1 Gulden.exe '-datadir=C:/datadir1 -port=7777"
"start /NODE 2 Gulden.exe '-datadir=C:/datadir2 -port=7778"

This is not quite equivalent to the hashrate a single correctly functioning instance would achieve but does manage to at least utilise all cores.

[mjmacleod]

Note that the mingw mailing list is to date not very receptive to the idea of actually patching mingw to be able to accept this.

https://sourceforge.net/p/mingw-w64/mailman/message/36893364/

Possible way(s) forward from here:

1. Patch mingw despite upstreams apparent reluctance. Maintain our own patched version and try and push upstream to accept the patch into the official version.
   This will mean some work to patch mingw, and some additional infastructure work to have our official builds make use of the patched mingw. We will also have to consider ongoing costs of reapplying the patch every time we want to upgrade to a newer mingw.

2. Switch to using mingw-w64 with native windows threads instead of "pthreads" - use https://github.com/meganz/mingw-std-threads or similar to deal with lack of std::thread support.
   The main problem here will be ensuring that all dependencies can build under this configuration.

3. Get the codebase compiling with MSVC, and use a setup like https://github.com/mstorsjo/msvc-wine for official builds - i.e. official builds on MSVC instead of mingw.
   This involves a fair bit of work to tackle various code compatibility issues to make everything compile nicely on MSVC, there is also some build system work to get MSVC builds working with our current build system and finally recipes to build all dependencies using MSVC.
   As a positive this will fix other issues we currently face - e.g. difficulty having our unity lib interact with electron on windows while it can do so on other platforms.