feat(provider-tck): emit a machine-readable conformance report

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to
<dir>/<name>.json against the report schema in the specification repository
(open-feature/spec#425, part of open-feature/spec#424).

Go is the language that needs this first. godog counts a capability-gated skip
in its passed tally, so a run that skipped five of twenty-nine scenarios prints
"29 scenarios (29 passed)". Appendix F is unambiguous that a scenario skipped
for an undeclared capability is reported as skipped with the reason and never as
passed, and the harness does say so in a separate log line -- but the headline
number still says something false, and the number is what gets read. pytest and
jest-cucumber report skips correctly, so this is the runner's property rather
than the suite's design.

The report does not fix godog's summary. It makes the summary stop mattering, by
recording every scenario's outcome individually so a consumer can check the rule
instead of trusting the runner to have applied it. The same run now reports
twenty-four passed and five not-declared, each with its reason.

Identity comes from revision.go, generated by sync_assets.go beside the embedded
artifacts. Generating both in one command is what keeps them honest: the CI check
regenerates and fails on any difference, so a revision disagreeing with the
artifacts beside it cannot be committed. The check is widened to cover the
generated file, which it would otherwise have missed. Both the commit and the
tree hash are recorded, the tree because it identifies the artifacts alone --
unchanged by unrelated edits elsewhere in the specification, so two runs of
identical artifacts agree even when pinned to different commits, and checkable
because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it.

Two smaller decisions. The provider is identified by the name it reports through
its own metadata, with Config.Name recorded as the configuration, because
Config.Name is chosen to read well in a failure message -- "flagd-rpc" -- and a
provider with two materially different modes produces two reports that are not
interchangeable. And how the backend was driven is read through an optional
interface rather than a new BackendControl method, so that adding it breaks no
existing implementation and a control that does not implement it simply omits
the field.

Emission is opt-in through the environment rather than through Config so that
producing a report is a property of the run and not of the code: CI sets it, a
local run does not, and no adopter changes a line to publish one.

The tests assert the property that motivated the work -- that no scenario the
capability gate stopped is ever reported as passed, and that every scenario is
accounted for exactly once, since a report that silently omitted what it skipped
would satisfy the first half while still misleading a reader. That test earned
its place immediately: it caught this emitter recording every skipped scenario
twice, the second time as passed, because godog does not deliver the before
hook's ErrSkip to the after hook.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Author: aepfli

Makefile

@@ -21,7 +21,7 @@ provider-tck-assets:
 	cd tools/provider-tck && go run ./sync_assets.go
 
 provider-tck-assets-check: provider-tck-assets
-	git diff --exit-code -- tools/provider-tck/pkg/tck/assets
+	git diff --exit-code -- tools/provider-tck/pkg/tck/assets tools/provider-tck/pkg/tck/revision.go
 
 test:
 	go list -f '{{.Dir}}/...' -m | xargs -I{} go test -v {}

tools/provider-tck/README.md

@@ -275,6 +275,62 @@ provider in a domain replaces and shuts down the previous one; a fresh domain pe
 leave every provider of the suite registered and running, which for a provider holding a network
 connection means leaking one connection per scenario.
 
+## Conformance reports
+
+Set `PROVIDER_TCK_REPORT_DIR` and each suite writes a machine-readable report of its run to
+`<dir>/<name>.json`, conforming to the [report schema][report-schema] in the specification.
+
+```console
+$ PROVIDER_TCK_REPORT_DIR=./reports go test ./...
+$ jq '.scenarios | group_by(.outcome) | map({(.[0].outcome): length}) | add' reports/in-memory.json
+{
+  "passed": 24,
+  "not-declared": 5
+}
+```
+
+It is an environment variable rather than a `Config` field so that emitting a report is a property
+of the run and not of the code: CI sets it, a developer running the suite locally does not, and no
+adopter changes a line to publish one. Unset means no report, which is not an error. Several suites
+in one test binary each write their own file, so flagd's two resolvers do not collide.
+
+### Why this exists in Go before the other languages
+
+Because Go is the language that needs it most. godog counts a capability-gated skip in its **passed**
+tally:
+
+```
+29 scenarios (29 passed)
+```
+
+Five of those twenty-nine did not run. Appendix F is unambiguous that a scenario skipped for an
+undeclared capability is reported as skipped with the reason and *never* as passed, and the harness
+does say so in a separate log line — but the headline number still says something false, and a
+number is what gets read. pytest and jest-cucumber both report skips correctly, so this is a property
+of the runner rather than of the suite's design.
+
+The report does not fix godog's summary. It makes the summary stop mattering, by recording the
+outcome of every scenario individually so that a consumer can check the rule instead of trusting the
+runner to have applied it. `reports/in-memory.json` above accounts for all twenty-nine scenarios and
+calls five of them `not-declared`, each with the reason.
+
+### What identifies a report
+
+`tck.specRevision` and `tck.assetsTree` come from [`revision.go`](./pkg/tck/revision.go), which
+`sync_assets.go` generates from the submodule alongside the embedded artifacts. Generating both in
+the same command is what keeps them honest: `make provider-tck-assets-check` regenerates and fails on
+any difference, so a revision that disagrees with the artifacts beside it cannot be committed.
+
+The tree hash is carried as well as the commit because it identifies the artifacts alone. It is
+unchanged by unrelated edits elsewhere in the specification, so two runs that executed identical
+artifacts report the same value even when pinned to different commits — and it is checkable, since
+`git rev-parse <specRevision>:specification/assets/provider-tck` must reproduce it.
+
+`provider.name` is what the provider reports through its own metadata, not `Config.Name`.
+`Config.Name` is chosen to read well in a failure message — `flagd-rpc` — which makes it the
+*configuration*, and it is reported as such. One provider with two materially different modes
+produces two reports that are not interchangeable.
+
 ## The self-tests
 
 Three suites run against providers from the SDK itself. They need no Docker and finish in
@@ -345,5 +401,6 @@ the SDK's provider rather than reimplementing it — every resolution decision i
 [appendix-b]: https://github.com/open-feature/spec/blob/main/specification/appendix-b-gherkin-suites.md
 [appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md
 [control-api]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/openapi/control-api.yaml
+[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json
 [spec]: https://github.com/open-feature/spec
 [tracking]: https://github.com/open-feature/spec/issues/417

tools/provider-tck/pkg/tck/capability.go

@@ -204,9 +204,13 @@ func (c Capability) Tag() string { return string(c) }
 // String implements fmt.Stringer.
 func (c Capability) String() string { return string(c) }
 
-// capabilityForTag maps a Gherkin tag onto the capability it gates, reporting
+// CapabilityForTag maps a Gherkin tag onto the capability it gates, reporting
 // whether the tag gates anything at all.
-func capabilityForTag(tag string) (Capability, bool) {
+//
+// Exported because a conformance report is read by things outside this package:
+// deciding whether a scenario was skipped legitimately means knowing which of
+// its tags gate a capability and which are merely organisational.
+func CapabilityForTag(tag string) (Capability, bool) {
 	for _, c := range allCapabilities {
 		if string(c) == tag {
 			return c, true
@@ -230,7 +234,7 @@ type capabilitySet map[Capability]struct{}
 func newCapabilitySet(caps []Capability) (capabilitySet, error) {
 	set := make(capabilitySet, len(caps))
 	for _, c := range caps {
-		if _, known := capabilityForTag(string(c)); !known {
+		if _, known := CapabilityForTag(string(c)); !known {
 			return nil, fmt.Errorf(
 				"unknown capability %q: capabilities are the constants declared in this package, one of %s",
 				c, formatCapabilities(AllCapabilities()))

tools/provider-tck/pkg/tck/inprocess.go

@@ -234,3 +234,10 @@ func (c *InProcessControl) ChangeFlag(context.Context) error {
 
 	return c.current.UpdateFlag(ChangingFlagKey, changingFlag(c.changingVariant))
 }
+
+// ControlAPI reports how this backend was driven.
+//
+// "in-process" is the narrow allowance the schema makes for a provider with no
+// backend. A report claiming it for a provider that has one should be treated
+// with suspicion, which is precisely why it is recorded rather than assumed.
+func (c *InProcessControl) ControlAPI() string { return "in-process" }