Can I run the build of tests in bazel?

[oharboe]

There's an inversion of control problem when running Chisel tests from Bazel as all the Verilator builds of a test is hidden deep down in Chisel and I end up with no caching as Bazel runs from a hermetic environment.

I'd like to build the test(elaboration and verilator) with Bazel and invoke the test from Bazel and run it in Chisel/scala.

Any pionters on how to do this?

Sorting out this problem would make tests enormously much faster... Especially for running large integration tests in the Chisel Simulator setup.

There's very crude knobs to adjust when building the Chisel tests #5051

[jackkoenig]

In short, no. I feel your pain also in a hermetic environment. There are a couple of directions that can make this work though:

1. Eschew Scala logic in testing in favor of Chisel APIs

Basically, just use Chisel to emit the design, then build your own testing infrastructure in Bazel that invokes Verilator and the simulation. You can use DPI in the flow via the Chisel DPI APIs [1].

There's some also some work related to this type of approach called "Inline Testing" (IMO slightly misnamed, it's more about attaching unit tests to a larger elaboration so that the unit test runs on the same Verilog) [2]. That similarly requires your own simulation infrastructure in Bazel to run the tests, but it also has ChiselSim/ScalaTest integration for running the tests in the other style.

2. Enhance ChiselSim to support this use case.

This is possible but non-trivial. Basically ChiselSim would need to be enhanced to support a "multi-phase" or "multi-invocation" API, you'd need to be able to:

A. Invoke ChiselSIm such that it elaborates the Verilog and generates all additional artifacts then stops.

You might want it to spit out some information about how it wants to (or would) invoke the simulator. I've thought about this and think it would be nice if it could spit out a Ninja build file because Ninja is very simple then a build tool like Bazel could have a plugin to ingest the Ninja file and run it in a hermetic environment.

B. Invoke ChiselSim pointing it to a directory containing the simulator binary so that it can set up the IPC connection and drive the simulation.

Because ChiselSim requires an actual object of the type of your design, it would have to re-evaluate it. This is not great for performance (but at least your hermetic builds are still cachable). This could be improved though with (C)

C. Make it such that ChiselSim execution doesn't require actually elaborating your design if it's already been elaborated. Put another way, support some sort of Chisel-level linking

There are a couple of ways this could work. The simplest (but least general) would be to change the ChiselSim API to use some sort of "box" type (e.g. use a Simulation[MyModule] rather than a MyModule directly). This box type could use some dynamic+macro magic to pretend to give you type-safe accessors to the ports of the module so you can peek and poke, but it actually at runtime validates that the type information is correct in communication with the simulator.

A better version of that would be to make Definition / Instance support true linking at the Chisel level--letting you create a Definition[MyModule] from some serialized .fir (and likely additional information). Then ChiselSim could suport testing an Instance[MyModule] like described above but it would be a bit more robust and also make it possible to link multiple Chisel elaborations, not just link the simulator.

[oharboe]

Thanks!

It is very helpful to know what is and isn't possible and what is and isn't happening...

An important use-case for us are peek-poke less(it is easier to express parallelism in Chisel than in Scala peek/poke) ChiselSim test, so basically we have:

      simulate(
        new ExecuteTestBench(i)
      ) { dut =>
        ResetProcedure.module(10)(dut)
        RunUntilFinished(maxCycles = 50000)(dut)
      }
    }

In Bazel it should be relatively straightforward to make a Verilator C++ wrapper that asserts reset for 10 cycles and waits for io_done to be asserted or time out after 50000 cycles.

For our tests that use peek/poke, we'd prefer to have the peek poke drive by Scala in ChiselSim though. We'd prefer to provide the .fir + executable from Bazel to ChiselSim, and to leave building to Bazel, but the running of the peek poke test could be in Chisel/Scala.

[oharboe]

Another idea: blackbox.

1. build .fir + .sv in Bazel + verilator .bin
2. mack a mock VERILATOR_BIN that copies the verilator file
3. set up a black box that has the same interface as the .sv top file. The Chisel code between 1. and 2. for the module top interface could be shared relatively easily and safely and VERILATOR_BIN copies the verilator generated executable

Yikes! :-)

[oharboe]

Some work in progress... The-OpenROAD-Project/bazel-orfs#405

[oharboe]

One more advantage of having the build happen in bazel is that bazel will handle provisioning of CPU resources. If I run Scala Chisel tests in parallel, then I can have many Verilator/firtool/elaboration activities happening, each using "all" CPUs in the system. This can lead to way, way too many processes being launched and all of the using a lot of memory, thus bringing CI servers/workstations to its knees.

[oharboe]

I was wondering how I would handle sub-tests. In Bazel & Scala Test there's a big test(normally one per file) and then smaller tests within a Scala test or a Scala test running within Bazel. Bazel allows passing a generic (a generic concept across all sorts of tests, Scala, Rust, Chisel, Python, etc.) test filter that allows tests to do a simple test filtering.

Sub-tests in ChiselSim puts up an additional problem: every test is individually run through elaboration, firetool and Verilator, without sharing any artifacts. This multiplies the build problem...

In Bazel and with peek poke less tests, I had the idea that I could instantiate all the sub-tests with a single TestTop module and run them all in parallel in simulation. Verilator would then re-use build results across these sub-tests. Filtering needs a bit of pondering, perhaps a simple solution will present itself...