How do we best use HWI (hardware interface)

[mimir-d]

context: HWI (source here) is a protobuf based cpp library that can be used to determine (machine scope) stable hardware location identifiers. Such identifiers could then be directly used in the output spec hardwareInfo location field. It has different backends that know how to interact with the hardware (eg. inband and oob) to get the data.

The current challenge is: how do we expand the usage of HWI such that it can be used in more diags. Some discussion has occured already in the weekly meetings, and following is a list of criteria:

• it should be able to support other languages, in a "pay for what you use" fashion. This could be bindings to the native compiled libs, or a completely separate entity such as a sidecar or service.
• since it comes with a build system (unfeasible and undesired to translate the codebase to other langs), the lifecycle of this entity (lib/service/etc) should be disconnected from the diag when possible.
• it needs to be as portable as possible (given that different users have different build infrastructures)
• anything else? @dframe-google @karagog

Given the above, there's 3 options:

1. keep as a lib and provide lang bindings; it always gets included in (and tied to) the diag lifecycle. This has the benefit of not needing stable versions of the lib, at the cost of complexity for the diag build (and enforcing a build system).
2. deploy as a sidecar; the diag talks to this sidecar by some defined protocol. Complexity comes in the protocol (+error reporting) and needing stable versions of the lib, however it opens up other usage patterns (more details later)
3. deploy as a service; also needs a protocol (on top of a socket transport). This implies there is some sort of infrastructure able to run this service. It may not be fully possible to get the hardware data since the only access is oob.

I think option 2 is most suited for the criteria, but with a couple of caveats so that we can get maximal portability.

• we can keep using the lib as compiled in for cpp based diags (or where it's acceptable for enforce the bazel build system)
• we should release stable (tagged) versions of the lib (likely independent codebase), that can accept a transport protocol on top, materializing into a sidecar
• since the protocol is unspecified, it can be a unix socket (for sidecar usage) or tcp socket (for service usage). This then covers both options 2 and 3

pros:

• nothing changes for the current usage of HWI, since the same code ties back into the cpp + bazel
• we support other languages by having a lang boundary between the diag and the HWI sidecar. This makes the sidecar lang agnostic and can later on be replaced by anything else satisfying the protocol intf.
• the diag and sidecar can have different lifetimes (pro but also a con), however it's expected that not a lot of sidecar versions would be in use at any point in time (and they should be backwards compat as per semver)
• the cardinality of the diags vs sidecar versions should be many-to-one (ideally, but certainly the sidecar versions would be a lot lower than the number of diags using them), which makes all diags smaller
• a permissions mechanism can be enforced, since the sidecar lives separately from the diag. Eg. the diag exercises the GPU but needs some privileged-derived identifier for the cpu or baseboard.
• the sidecar/service can be built independently of the diag, which in some cases is needed for audit purposes. Eg. the diag is python so all the code is available, but the sidecar is compiled cpp which goes through a different build (and since we dont have hermetic builds, needs to be built from source internally to said company entity)

cons:

• need a bit of work to extract the codebase into a separate lib, and then hook back in this module into ocpdiag cpp lib
• needs a protocol spec between the diag and sidecar; not completely trivial but we should be able to just have the main operation in the protocol (eg. "give me the hardware identifiers" -> "list of")
• would require that the sidecar is installed along with the diag. This can be handled in the diag protocol layer, either refusing to run or specifying why the DUT data is not available.

Conclusion is that with a bit of change in the presentation of the lib, we could actually support all 3 usage patterns, with larger benefits than the impl/maintenance cost.

@dframe-google could you also attach the comparison doc you had internally? And maybe more details on HWI itself.
Anything I missed? Opinions? Thanks

[dmbsocram]

I have some questions on versioning of the HWI. How do diags specify HWI version requirements? How do we handle the case were different diags require different versions of the HWI?

[karagog]

This would be a concern with either the "side-car" approach, as well as a pre-built lib approach (but not an issue for the way we currently use it at Google, which is to package the HWI inside the diag itself, so the versions are tightly coupled). IME the API for the HWI is considerably less stable than the Meltan spec, as we might add new API methods freely as new requirements come along. We may have to consider a versioning system with major and minor version numbers to indicate API-breaking changes.

Note that there are other versioning considerations as well, like the fact that the HWI may depend on particular versions of Redfish, for example. It seems each version of the HWI must specify what versions of Redfish, etc... would be supported.

[mimir-d]

if i think about it, there's actually 2 versions to discuss; one is the hwi package itself (either sidecar or service; basically anything non-coupled with the diag itself) and one is the protocol to talk to hwi. A package version may support any number of protocol versions.
That said, semver is a standard way to approach this and as long as we keep things reasonably backwards compat (new api methods are usually ok, removing/changing is problematic), this mechanism should be feasible given the advantages above.

[karagog]

One concern I want to call out on the matter of open-sourcing the HWI is that we (Google) currently have high velocity when making changes to it, and I worry about this slowing down if it requires input from so many external parties. IMO we already kind of have this attitude that we may as well not even bother with changing the OCPDiag spec because of how much overhead would be involved to wrangle public opinion. I worry about this same kind of "helplessness" to control our own destiny if every change needs to run through an external committee first.

Have we thought about the support model for this HWI package, such as who would be in control of major decisions regarding it? Or are we assuming that we will cede control of it to the open source community?