refactor(provider-tck): emit a conformance report, with results in Cucumber Messages

[aepfli]

Emits a machine-readable conformance report for each suite run, when
TCK_REPORT_DIR is set. Unset means no report, which is the default and is not
an error. It gates an output, not the run: the suite runs either way, and
which target ran it is make tck's business rather than an environment
variable's.

Tracking: open-feature/spec#417. Appendix F: open-feature/spec#423. Report
format: open-feature/spec#424. Schema: open-feature/spec#425.

What a run writes

Two files. An envelope at <dir>/<name>.json, conforming to the schema in
open-feature/spec#425, and the results it references at <dir>/<name>.ndjson,
which is a Cucumber Messages stream.

{
  "schemaVersion": "1",
  "provider": { "name": "InMemoryProvider", "language": "go", "configuration": "in-memory" },
  "sdk": { "name": "github.com/open-feature/go-sdk", "version": "v1.18.0" },
  "tck": {
    "implementation": "go-sdk-contrib/tools/tck",
    "version": "v0.1.0",
    "specRevision": "4cab032043f9b2335a023310b1010b20d7a7d0d9"
  },
  "backend": {
    "description": "the Go SDK's memprovider.InMemoryProvider, rebuilt per scenario",
    "controlApi": "in-process"
  },
  "declaration": { "declared": ["@events", "@numeric-coercion", "@object"] },
  "results": {
    "format": "cucumber-messages",
    "location": "in-memory.ndjson",
    "digest": "sha256:4754d458ac5a1a137080082b7947f8f1eafc5df9f6d54551440da9f8ac6a0dce"
  }
}

The envelope identifies what was tested and what the provider claims. It does
not contain the results. A Messages stream carries the feature sources and so is
far larger than the envelope, and a consumer deciding whether it cares about a
report should not have to fetch a whole run to find out. results.digest covers
the payload byte for byte.

Why the results are a standard format

This PR previously defined its own: a per-scenario list, a four-value outcome
enum, and a field naming the Examples row an entry came from. All three are
gone. The per-scenario outcome, the tags, the executed feature source and the
identity of a Scenario Outline row are already specified by Cucumber Messages,
which is maintained, cross-language, schema'd, and emitted natively by
cucumber-jvm. Restating them here meant a second format to version, and two
places for the same fact to disagree.

Reading the results needs no bespoke tooling: every scenario is a pickle,
every result is a testCase with one testStepFinished per step, and a
scenario's outcome is the most severe of its step results, which is how Cucumber
itself derives it.

example is removed because Messages already carries row identity. A
pickle's astNodeIds end with the id of the Examples TableRow it was expanded
from, and the stream carries the gherkinDocument those ids belong to, so the
row's cells are recoverable from the stream alone. Four implementations had each
reinvented that field independently, one of them by reverse-engineering how its
runner maps a pickle back to a table row.

Removing it removed a coupling too. Recovering the row from a *godog.Scenario
meant reparsing the embedded feature files and reproducing the AST node ids
godog assigns, which come from a counter godog shares across the files it
parses, so reproducing them meant reproducing godog's whole parse. That is what
pkg/tck/examples.go did, and it is deleted. A formatter is handed the same
gherkinDocument godog compiled the pickles from, so the ids agree by
construction rather than by imitation.

assetsTree is removed because Messages carries the executed source. The
tree hash existed so a consumer could tell which questions a report answers
without trusting the recorded commit. The stream's source messages answer that
with the feature text rather than with a hash of it, and unlike the hash cannot
be asserted wrongly, because it is the input godog parsed.

What stays OpenFeature-specific, and why

The declaration. declaration.declared is an input to reading the results
rather than a summary of them, which is why it cannot be derived from the
payload. A SKIPPED scenario says the question was not put to this provider;
only the declaration says whether that is because the provider declines the
capability. Given the declaration and a scenario's tags, which the stream
carries, the reason for a skip follows without being transported per scenario.

The tested subject. No standard results format has a slot for the provider
under test: Messages records the runner, the runtime and the OS, not the
subject. So provider, sdk, tck and backend stay in the envelope.

@strict-numeric-typing is now @numeric-coercion, and the rule is corrected

tck.StrictNumericTyping becomes tck.NumericCoercion and its tag becomes
@numeric-coercion. The spec submodule moved to dc4d7ae8 for this change and the embedded assets
were re-synced from it, so SpecRevision changed too. It has since advanced to 4cab0320, which is
what the envelope above records; the assets are byte-identical across that range.

The old name was chosen for a stricter rule than the specification wants. flagd
is implementing an accepted numeric coercion ADR
(open-feature/flagd#1996) whose rule is that coercion is permitted when
lossless and must fail with TYPE_MISMATCH only when information would be
lost: 10.0 requested as an integer succeeds, 0.5 does not. Appendix F
previously said "does not coerce between integer and float", which forbids the
case the ADR requires to work. flagd's own testbed is gaining
@numeric-coercion scenarios as part of that work, so keeping a second name for
one property was drift, not caution.

When this was written the tag had one scenario, asking about 0.5 — the lossy
half — and this section recorded "the lossless case has no scenario" as an open
gap. That gap is closed. The canonical set gained integral-float-flag, and
errors.feature now carries three @numeric-coercion scenarios: the lossy one,
10.0 requested as an integer, and 10 requested as a float. A provider
declaring the tag must satisfy all three, which is what stops the shortcut of
rejecting every float and calling it strictness — the shortcut two of the four
OFREP adoptions turned out to be taking.

One gap does remain, and Appendix F records it rather than closing it here:

• Accessor width is modelled, and what remains open is narrower than it looks. The ADR
  distinguishes a 64-bit integer accessor from a 32-bit one, and flagd's testbed tags the 32-bit-only
  scenarios @int32-bounded. Appendix F draws the same boundary from the other side: every language's
  accessor can ask for 2^31 − 1, so that precision scenario is untagged and mandatory; only some
  can ask for 2^53 − 1, so that one carries @large-integers, which a provider on a 32-bit
  accessor leaves undeclared. Mandatory-versus-gated instead of two tags, and no tag needed for the
  common case.

  What is genuinely open is the negative half: nothing asserts what a 32-bit-accessor provider owes
  when asked for a value it cannot represent. The scenario simply skips, so a provider that silently
  truncates passes exactly like one that returns TYPE_MISMATCH. That is deliberate rather than
  overlooked — spec#430 has not settled what a
  provider owes a value that does not fit the requested accessor, and asserting it here would be this
  suite inventing a rule the specification does not have.

The capability's doc comment says so.

A reserved capability can no longer be declared

@caching is reserved: part of the vocabulary, carried by no scenario. It was
not the only one when this was written — @targeting was reserved too, and
stopped being so at spec 26362f85, when scenarios started carrying it. That is
the shape a reserved tag is meant to have: a placeholder that either grows
scenarios or is removed, never a claim a report can carry indefinitely. Appendix F now states plainly that such a tag must not be declared and
must not appear in a report's declaration — nothing carries it, so declaring it
cannot be verified, cannot even produce a skip, and tells a reader of the report
only that something was claimed and nothing examined.

This was a live defect rather than a hypothetical one. A Java conformance report
asserts both tags as declared, not by anyone's decision but because that
adoption declares "every capability except X" and collects every reserved tag on
the way past. AllCapabilities() had the same shape, and it is the default when
Config.Capabilities is nil.

• The reserved set is named once, beside the constants, and exposed as
  Capability.IsReserved.
• AllCapabilities() omits it, which also fixes the nil default. Its doc
  comment says what it now returns and why.
• An adopter who names a reserved capability outright is failed, not warned.
  The check lives in newCapabilitySet, which Config.validate calls, so the
  run stops before any scenario with the same message shape as any other
  configuration problem. Warning would leave the claim in a published report,
  which is the outcome the rule exists to prevent, and the fix is to delete one
  line.
• Putting the check there rather than only in Config.validate is deliberate:
  the capability set is the only thing declaration.declared is built from, so
  a reserved tag now has no path into a report at all, and there is no second
  route that could drift from the rule.

The Messages formatter

godog 0.15.1 has no Messages formatter. It registers cucumber (the legacy
relishapp JSON), events, junit, pretty and progress. So
pkg/tck/messages.go is one, registered through the public godog.Format
plugin interface and consuming the formatter event stream. Nothing in it
implements the Messages specification: the types come from
github.com/cucumber/messages/go/v21, which this module already depended on. It
is a candidate for contributing upstream to godog.

godog's JUnit output was not a usable fallback: a capability-gated skip comes
out as skipped="0" on the suite, in a non-standard <error type="skipped">
element, with the step text where the skip reason should be.

That is not a coincidence, and it is the reason the events are the right source.
For the first step of a gated scenario godog inserts a FAILED step result into
its internal storage and calls Skipped on the formatter. The built-in
formatters read storage, which is where the malformed JUnit comes from; a
formatter built on the events sees one SKIPPED. External formatters cannot
read storage anyway, since SetStorage takes an internal type, which here is a
feature.

Two limits of the formatter interface, both worked around and both documented in
the file:

• No scenario-finished event. The interface has TestRunStarted, Feature,
  Pickle, the per-step results and Summary, and nothing that fires when a
  scenario ends. The stream is therefore accumulated and written at Summary
  rather than streamed, which also lets the envelopes be emitted in the
  conventional order. It assumes serial execution, which this suite already
  enforces for unrelated reasons.
• No reason for a skip. Skipped(pickle, step, definition) carries no
  error, so the capability that gated a scenario is unrecoverable from the
  events. The reason comes from the capability gate itself and lands in
  TestStepResult.message. This is the one part of the formatter that is
  specific to this suite rather than general.

The skip is truthful

This is the whole reason the report exists. godog counts a capability-gated skip
in its passed tally:

29 scenarios (29 passed)

Five of those twenty-nine did not run. Appendix F is unambiguous that such a
scenario is reported as skipped with the reason and never as passed.

The stream for that same run reports 24 PASSED and 5 SKIPPED, accounts for
all 29 scenarios exactly once, and gives every skip a reason naming the
capability that caused it. At step level, 182 PASSED and 26 SKIPPED.

Three schema changes since this was opened

All three are in spec#425 and
this branch emits against them.

• backend.controlApi is required. A provider with no backend still had its
  flag state manipulated somehow, and which of the two ways that was is the
  single most important thing a reader needs to know what the results are worth.
  The old shape made the whole backend block omissible, which made the one
  value most worth knowing — in-process, the case the enum exists for — the one
  that could never appear. ReportBackend.ControlAPI is never empty: the control
  states it, nothing infers it.
• A known deviation's summary is required, and its issue link is not. A
  deviation with no prose is a line of tags that tells a reader nothing; a
  deviation with no filed issue is an honest state to be in, and requiring one
  encourages filing a placeholder.
• The declaration no longer claims to explain every skip. It explains the
  skips that come from an undeclared capability. It does not explain a skip the
  implementation refuses — @large-integers in Java, @numeric-coercion in
  JavaScript — which is a property of the SDK rather than of the provider, and a
  reader who conflated the two would read a language's limit as a provider's
  decision.

Verification

• gofmt, go build ./..., go vet ./... clean; package tests pass, including
  under -race.
• Three suites emitted real reports. All three envelopes validate against the
  schema with a Draft 2020-12 validator, and every results.digest matches its
  payload.
• Each stream, 543 messages (meta, 4 source, 4 gherkinDocument, 29
  pickle, 29 each of testCase/testCaseStarted/testCaseFinished, 208 each
  of testStepStarted/testStepFinished, testRunStarted,
  testRunFinished), validates against the published Cucumber Messages
  v21.0.1 JSON schemas with zero errors — and also against 24.1.0, a
  newer major than the types that produced it, so the stream is readable by a
  current cucumber toolchain and not only by the version it was emitted with.
• The eleven rows of "Requesting the wrong type returns the code default" in
  errors.feature resolve to eleven distinct astNodeIds and eleven distinct
  Examples rows, recovered from the stream's own gherkinDocument.

For the two changes above:

• make provider-tck-assets-check passes after the submodule bump. The synced
  assets contain @numeric-coercion and no occurrence of
  @strict-numeric-typing.
• Outcome counts over the three streams are unchanged in shape:
  controllable-in-memory 26 PASSED / 3 SKIPPED, in-memory 24 / 5,
  multi-provider 25 / 4. The renamed scenario, "A float flag is not silently
  narrowed to an integer", is reported PASSED under @numeric-coercion in all
  three, and every gated skip is still SKIPPED with a reason naming its
  capability.
• declaration.declared in all three envelopes contains neither @targeting
  nor @caching.
• Four tests cover the reserved rule: AllCapabilities() excludes exactly the
  reserved set and nothing else, the nil default excludes it, Config.validate
  rejects each reserved capability by name, and the declaration built by
  buildReport from the default set carries no reserved tag. Removing the
  exclusion from AllCapabilities() fails three of them.

The submodule bump also picks up two unrelated spec changes between dfa16586
and dc4d7ae8: the lifecycle readiness scenario is renamed to "A provider that
successfully initializes becomes ready", and the control API's POST /start
gains the requirement that it not return until the seeded flag state is being
served. Nothing in this module referenced the old scenario name, and no Go code
implements /start.

Tests were rewritten accordingly: the assertions about scenarios[], the four
outcomes and example are gone, replaced by assertions over the Messages
stream. The two properties that matter are kept and asserted directly, that no
gated scenario is ever reported as passed and that every scenario is accounted
for exactly once. The internal tests drive the formatter through godog's event
sequence to cover the FAILED path no passing suite reaches.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}