Use Nix derivations to run Nimbus software

[jakubgs]

We've had weird failures in the past when building Nimbus software locally, ones like this:

error: cannot lock ref 'refs/remotes/origin/test/lsquic/dirs': 'refs/remotes/origin/test' exists; cannot create 'refs/remotes/origin/test/lsquic/dirs'
 ! [new branch]          test/lsquic/dirs       -> origin/test/lsquic/dirs  (unable to update local ref)
Fetching submodule vendor/nimbus-build-system
From https://github.com/status-im/nimbus-build-system
   67f3958..4b26def  master     -> origin/master
Errors during submodule fetch:
        vendor/nim-libp2p
update-beacon-node-hoodi-testing.service: Main process exited, code=exited, status=1/FAILURE

Which block testing of new software versions on the fleets and require manual intervention.

Ever since the NBS module has been properly implemented in Nix in nimbus-eth2 and nimbus-eth1:

• nix: use Nix NBS flake and add derivations nimbus-eth1#4102
• nix: use Nix Flake from NBS repo to provide Nim nimbus-eth2#8185

We can now drop the ugly build.sh script in favor of simply calling:

nix build 'git+https://github.com/status-im/nimbus-eth2?ref=testing&submodules=1#beacon_node'

To build latest testing branch commit. We should test this approach on a single Hoodi host to see how well it works.

In addition, we should implement a Consul check that confirms via REST API call that the current running node has a recent enough commit from its respective build branch, and if it's for example older than 1 day mark it as warning and more than 2 days as critical.

[jakubgs]

For extra credit we could add a nightly CI job that builds Nix derivations and pushes them to our Nix cache. Then if we add our cache to flake.nix files we can even avoid rebuilding most of the time.

Not strictly necessary but a nice optimization.

[jakubgs]

One thing worth checking is if Nix isn't sometimes stupid and using ref=testing will work when a fresh commit is pushed, of if it might use previously fetched values. There are cases where purging ~/.cache/nix folder is necessary to force it to fetch newest.

[markoburcul]

To build it we need to use the trick with git+https and not github, so the command is:
nix build 'git+https://github.com/status-im/nimbus-eth2?submodules=1&ref=testing#beacon_node'

[markoburcul]

I've installed nix using our infra-role-nix to all of the Linux hosts in infra-nimbus. All of the BN's and VC's that are running testing and unstable branch now use Nix to build the executable and the commit drift is handled by the script and consul checks.
After the nix related fixes land into stable, I will update the rest of the fleet to use the same mechanism for update. Still need to check what is the situation with libp2p branch.
Next step is to add the nightly nimbus-eth2 pipeline which will build all of these relevant branches for BN and VC and push them to our nix cache. After that we can define in the nimbus-eth2 flake the cache and thus the builds should be much faster.