Research alternative orchestrator for nimbus-eth1 bechmarks

[siddarthkay]

Problem Statement

Currently nimbus-eth1 benchmark is running as an independent systemd service which just keeps looking for newer commits in nimbus-eth1 github repo and runs the benchmark for it.
This process is not smart enough to take into account about missed benchmarks in the past and if theres no new commits this service just does nothing and restarts 30 seconds later.

We currently also utilise bench-01 only for short benchmark runs and bench-02 for long benchmark runs.
There comes a time when the host has no work to do because short benchmark is supposed to finish quite faster.

The setup is also not scaleable and if we add new hosts to the setup we will have to find a way to orchestrate a queue and decide what job the new host should run : short or long benchmark.

Action Items

• Implement a Queue based system which keeps track of pending jobs. In this case we can pick Jenkins.
  Its easier to visualise and manage the queue in Jenkins.

  • Jenkins also allows us to distribute jobs among agents running on different hosts.
  • Need to do some research on how many items can live in a Jenkins queue, we can also make a static list for our current queue and Jenkins can refer to this list for scheduling new jobs.

• Update report generation script to move on from timestamp based approach to git history based comparison.

  • Currently we compare latest benchmark against last run and generate a delta, to figure out if the benchmark got faster or slower. Out of order runs mess this up and give us incorrect delta for such cases.

• We need to decide on the logic for Queue population by Job Type: I would assume that completing all short benchmarks would have higher priority over completing all long benchmarks, but this needs to be discussed and agreed upon with @arnetheduck and @jakubgs

[siddarthkay]

This Queue could be a bit smart and look like this :

Priority Order:

1. New commits (short) -> immediate feedback on regressions
2. New commits (long) -> complete coverage for new work
3. Missed commits (short) -> fill gaps in short coverage
4. Missed commits (long) -> fill gaps in long coverage (lowest priority)

This "Queue Populator" job could run hourly and:

1. Fetch latest commits from nimbus-eth1
2. Check which have short/long benchmarks
3. Add missing ones to the queue with appropriate priority

[jakubgs]

I would try to avoid writing queue management logic and use existing solutions like Jenkins plugins for queue priority:

• https://plugins.jenkins.io/PrioritySorter/
• https://plugins.jenkins.io/simple-priority-sorter/

There's probably more. But we could certainly have a meta-job that spawns backlog filling jobs for a commit range.