feat: add a schema for machine-readable provider conformance reports by aepfli · Pull Request #425 · open-feature/spec

aepfli · 2026-08-24T14:08:44Z

Adds a JSON Schema for the envelope of one provider conformance run: what was tested, what the provider claims, and where the executed results are. The results themselves are Cucumber Messages, not a format this specification defines.

Stacked on #423 (Appendix F), which defines the suite this reports on. Review that one first; this PR is only the report envelope.

Phase 1 of #424, which is the design discussion. Collection and trust — how OpenFeature gets reports from providers it does not host, and how much a self-published report should be believed — stay open on the issue and are deliberately not addressed here.

What changed since the first draft

The first draft defined its own per-scenario result list, a four-value outcome enum, and a field identifying which Scenario Outline row an entry came from. All three already exist in Cucumber Messages, which is maintained, cross-language, JSON-schema'd, and emitted natively by cucumber-jvm and godog. Defining them again would have created a second format to maintain and version, and two places for the same fact to disagree. That was the point of @toddbaert's comment on #424, and this PR now follows it.

The outline-row field was the clearest case for dropping ours: it was added days ago, and four implementations each reinvented it independently — one by reverse-engineering how its runner maps a pickle back to a table row. Messages has carried that identity as AST node ids all along.

Messages is also stronger than what it replaces. It carries the executed feature source, so a consumer can verify which questions were asked rather than trusting a recorded revision.

What the envelope still has to carry

The declaration. Which capabilities the provider claims is an input to reading the results, not a summary of them. A skipped scenario says the question was not put; only the declaration says whether that is because the provider declines the capability. Given the declaration and a scenario's tags, the reason for a skip follows, so it no longer needs transporting per scenario. A capability that no executed scenario carries must not be declared, because it cannot produce a skip and listing it invites a reader to believe it was verified.

notApplicable, kept separate from undeclared. Collapsing them misrepresents whole languages: @numeric-coercion is unsatisfiable in JavaScript because the language has no integer type, and reporting that as an undeclared choice would show every JavaScript provider as missing something none of them can have.

The tested subject and its provenance. No standard results format has a slot for "the provider under test": Messages records the runtime and OS, not the subject. configuration matters because one provider can produce several non-interchangeable reports — flagd's RPC and in-process resolvers differ in whether they emit PROVIDER_STALE.

specRevision, corroborated rather than merely asserted for the feature files, because the Messages stream carries the executed source. For the flag set and the control API it stands alone. An earlier draft carried a git tree hash of the asset directory to close that gap; it was dropped because every implementation either reads the assets from the submodule directly or regenerates and diffs its copy in CI, so drift is caught where it happens.

knownDeviations, distinguishing "chose not to implement an optional feature" from "has a known bug, tracked here". Three of the four implementations needed it before it existed: flagd narrows floats in two languages, Python marks one scenario as an expected failure against an SDK bug, and the JavaScript multi-provider flattens child error codes to GENERAL.

No aggregate score. No percentage, no "27/29". A single number invites comparing providers that declared different capability sets, which is exactly the comparison the tag vocabulary exists to prevent.

Results are referenced, not inlined

A Messages stream carries the feature sources and is far larger than this envelope, and a consumer deciding whether it cares about a report should not have to fetch a whole run to find out. The envelope carries format, formatVersion, location and an optional digest. Two lossier formats are admitted for runners that cannot emit Messages yet, with what each loses stated: cucumber-json identifies outline rows only by line number, and junit-xml carries no tags at all.

One thing a consumer of a Messages stream needs to know: testCaseFinished carries no status. A scenario's outcome is the most severe testStepResult among its steps, hooks included. That is what makes a capability skip visible — a gate aborting in a before-hook produces a SKIPPED hook result outranking the PASSED steps it prevented.

What is not here

Emitters that produce this envelope; the existing ones in Go, Python and JavaScript wrote the earlier draft format and will be updated once this shape is agreed. Also still open on #424: whether a report should be published inside the provider's released artifact, and how strongly the tested subject revision must be identified.

Closes nothing; #424 stays open for collection and trust.

coderabbitai · 2026-08-24T14:08:52Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

_{Comment @coderabbitai help to get the list of available commands.}

@object

A scenario entry in the conformance report was identified by feature and name. Every row of a Scenario Outline shares one name, so the type-mismatch matrix in errors.feature produced eleven entries that differed only in durationMs. If one row failed and ten passed, the report could not say which failed, and a consumer keying on feature and name kept whichever row it read last. Per the report schema, a scenarioResult now carries `example`: the Examples row it came from, keyed by column header, with the cells verbatim as strings. Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says what the table said. It is present only for outline rows and omitted otherwise. godog hands a hook an already-expanded pickle, whose step text has the parameters substituted in and whose row is otherwise gone. What survives is AstNodeIds, whose last entry is the id of the Examples TableRow. The row is therefore recovered by parsing the embedded feature files a second time and indexing every TableRow by that id. Those ids come from a counter godog creates once per run and shares across the files it parses, so reproducing them means reproducing godog's parse -- same files, same order, pickle compilation in between. That coupling is not left to be trusted: a pickle that came from an outline and did not resolve fails the run, because quietly returning to the ambiguity this field exists to remove is worse than a build failure. The capability gate records its outcome before a scenario starts, so it fills the field in too. Four skipped rows of the @object outline are as ambiguous as four failed ones. The gate's own bookkeeping is keyed by pickle id rather than by scenario name for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row and the rows that did run would have vanished from the report. gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes. Verified against the schema on open-feature/spec#425 with a Draft 2020-12 validator: every report the self-tests emit validates, the eleven matrix rows carry eleven distinct examples matching the feature file, and (feature, name, example) is unique across every scenario in a report. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

The emitter defined its own per-scenario result list: a four-value outcome enum, a tag list, a reason, and a field naming which Scenario Outline row an entry came from. All of it already exists in Cucumber Messages, which is maintained, cross-language, schema'd, and emitted natively by cucumber-jvm. The report schema was reshaped to reference a Messages payload rather than define one (open-feature/spec#425); this follows it. A run now writes two files per suite: <name>.json, the envelope, and <name>.ndjson, the results it points at, with results.digest over the exact bytes written. Deleted, because Messages carries them: scenarios[] - now TestCase/TestCaseStarted/TestStepFinished/TestCaseFinished. the outcome enum - Cucumber's own seven statuses. The declared/not-applicable distinction was never a property of the run: it follows from the declaration and the scenario's tags, so it is stated once in the envelope instead of once per scenario. example - a pickle's astNodeIds are [scenario id, table row id], and the row id resolves in the GherkinDocument to the cells the feature file wrote. Four implementations were each reinventing this field by hand. tck.assetsTree - the payload carries the executed feature Source verbatim, which answers "did two runs ask the same questions" directly rather than by proxy. Two things Messages cannot carry, so they stay. The declaration is an input to reading the results, not a summary of them. And no standard results format has a slot for the tested subject: Messages records the runtime and the OS, not what was being asked about. pytest-bdd emits no Messages -- it ships the legacy Cucumber JSON format -- so messages.py assembles the stream. Two dependencies, each doing the half it owns: cucumber-messages, the official Python types from the protocol's own repository, for the execution messages; gherkin-official, already a transitive dependency of pytest-bdd, for the gherkinDocument and pickle payloads, which are used as it produces them rather than round-tripped through another representation. The feature files are parsed again because pytest-bdd's own dataclasses drop the AST node ids a pickle refers to. Step results come from pytest-bdd's step hooks rather than from the scenario's verdict, because a stream that marked all eight steps of a scenario failed would be saying something untrue about the seven that passed and the ones never reached. Each test case also carries a before- and after-hook TestStep: pytest runs three phases and only the middle one executes steps, so that is where a capability skip's reason and a teardown failure belong. A verdict no step accounts for -- a strict xfail that passes -- is attached to the after-hook, so it survives a consumer computing the test case's status as the worst of its steps. An expected failure is still a failure in the payload. The acknowledgement moved to the envelope's knownDeviations, declared by TckConfig.known_deviations, where it records the gap without softening the result. TckConfig also gains not_applicable, for a capability that cannot hold rather than one the provider declines. Verified locally; CI does not run on this branch, which targets the report branch rather than main. Both suites' envelopes validate against the reshaped schema with a Draft 2020-12 validator and their digests match; both streams validate clean against the Cucumber Messages JSON schema at v34.2.0 (661 messages each, zero errors). The stream accounts for all 29 collected scenarios; the five the capability gate stopped are SKIPPED for every step, none PASSED, and the one row the SDK fails is FAILED while pytest exits zero. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Go is the language that needs this first. godog counts a capability-gated skip in its passed tally, so a run that skipped five of twenty-nine scenarios prints "29 scenarios (29 passed)". Appendix F is unambiguous that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and the harness does say so in a separate log line -- but the headline number still says something false, and the number is what gets read. pytest and jest-cucumber report skips correctly, so this is the runner's property rather than the suite's design. The report does not fix godog's summary. It makes the summary stop mattering, by recording every scenario's outcome individually so a consumer can check the rule instead of trusting the runner to have applied it. The same run now reports twenty-four passed and five not-declared, each with its reason. Identity comes from revision.go, generated by sync_assets.go beside the embedded artifacts. Generating both in one command is what keeps them honest: the CI check regenerates and fails on any difference, so a revision disagreeing with the artifacts beside it cannot be committed. The check is widened to cover the generated file, which it would otherwise have missed. Both the commit and the tree hash are recorded, the tree because it identifies the artifacts alone -- unchanged by unrelated edits elsewhere in the specification, so two runs of identical artifacts agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with Config.Name recorded as the configuration, because Config.Name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read through an optional interface rather than a new BackendControl method, so that adding it breaks no existing implementation and a control that does not implement it simply omits the field. Emission is opt-in through the environment rather than through Config so that producing a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. The tests assert the property that motivated the work -- that no scenario the capability gate stopped is ever reported as passed, and that every scenario is accounted for exactly once, since a report that silently omitted what it skipped would satisfy the first half while still misleading a reader. That test earned its place immediately: it caught this emitter recording every skipped scenario twice, the second time as passed, because godog does not deliver the before hook's ErrSkip to the after hook. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

@object

A scenario entry in the conformance report was identified by feature and name. Every row of a Scenario Outline shares one name, so the type-mismatch matrix in errors.feature produced eleven entries that differed only in durationMs. If one row failed and ten passed, the report could not say which failed, and a consumer keying on feature and name kept whichever row it read last. Per the report schema, a scenarioResult now carries `example`: the Examples row it came from, keyed by column header, with the cells verbatim as strings. Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says what the table said. It is present only for outline rows and omitted otherwise. godog hands a hook an already-expanded pickle, whose step text has the parameters substituted in and whose row is otherwise gone. What survives is AstNodeIds, whose last entry is the id of the Examples TableRow. The row is therefore recovered by parsing the embedded feature files a second time and indexing every TableRow by that id. Those ids come from a counter godog creates once per run and shares across the files it parses, so reproducing them means reproducing godog's parse -- same files, same order, pickle compilation in between. That coupling is not left to be trusted: a pickle that came from an outline and did not resolve fails the run, because quietly returning to the ambiguity this field exists to remove is worse than a build failure. The capability gate records its outcome before a scenario starts, so it fills the field in too. Four skipped rows of the @object outline are as ambiguous as four failed ones. The gate's own bookkeeping is keyed by pickle id rather than by scenario name for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row and the rows that did run would have vanished from the report. gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes. Verified against the schema on open-feature/spec#425 with a Draft 2020-12 validator: every report the self-tests emit validates, the eleven matrix rows carry eleven distinct examples matching the feature file, and (feature, name, example) is unique across every scenario in a report. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Unset means no report, which is the default and is not an error. An environment variable rather than a TckConfig field, so that emitting a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. Several suites in one pytest session each write their own file, so flagd's two resolvers would not collide. The load-bearing part is the per-scenario list. Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and nothing downstream can check that against a summary line. Recording every scenario's outcome individually makes the rule checkable by the consumer instead of dependent on the runner. It is also required to be complete, because a document that quietly dropped what it skipped would satisfy the letter of the rule and still mislead whoever read it. pytest, unlike godog, reports a skip honestly -- so the interesting divergence here is elsewhere. The one scenario the Python SDK cannot satisfy is marked xfail, so the run finishes green; the provider still did not satisfy it, and the document says failed with the reason. An expected failure is a recorded deviation, not an excused one. Scenarios are therefore enumerated at collection and resolved at the end of the session rather than as fixtures run, which is also what keeps a scenario skipped by a marker -- whose fixtures never run at all -- from vanishing from the document. Identity comes from spec_revision.json, generated by hatch_build_sync.py beside the copied assets and force-included into the wheel. It has to be captured at build time: the submodule that knows the answer is not in the distribution, so an installed copy has nothing left to ask. A build that cannot reach git -- an unpacked sdist -- warns and records "unknown" rather than inventing a commit. Both the commit and the tree hash are recorded, the tree because it identifies the assets alone: unchanged by unrelated edits elsewhere in the specification, so two runs of identical assets agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with TckConfig.name recorded as the configuration, because TckConfig.name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read off an optional control_api property rather than added to the BackendControl protocol, so that adding it leaves every existing control complete and one that stays quiet simply omits the field. The tests assert the two properties a consumer is entitled to assume -- that no scenario the capability gate stopped is ever reported as passed, and that every collected scenario appears exactly once, counted against pytest's own collection rather than against a number written down beside it. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

The emitter defined its own per-scenario result list: a four-value outcome enum, a tag list, a reason, and a field naming which Scenario Outline row an entry came from. All of it already exists in Cucumber Messages, which is maintained, cross-language, schema'd, and emitted natively by cucumber-jvm. The report schema was reshaped to reference a Messages payload rather than define one (open-feature/spec#425); this follows it. A run now writes two files per suite: <name>.json, the envelope, and <name>.ndjson, the results it points at, with results.digest over the exact bytes written. Deleted, because Messages carries them: scenarios[] - now TestCase/TestCaseStarted/TestStepFinished/TestCaseFinished. the outcome enum - Cucumber's own seven statuses. The declared/not-applicable distinction was never a property of the run: it follows from the declaration and the scenario's tags, so it is stated once in the envelope instead of once per scenario. example - a pickle's astNodeIds are [scenario id, table row id], and the row id resolves in the GherkinDocument to the cells the feature file wrote. Four implementations were each reinventing this field by hand. tck.assetsTree - the payload carries the executed feature Source verbatim, which answers "did two runs ask the same questions" directly rather than by proxy. Two things Messages cannot carry, so they stay. The declaration is an input to reading the results, not a summary of them. And no standard results format has a slot for the tested subject: Messages records the runtime and the OS, not what was being asked about. pytest-bdd emits no Messages -- it ships the legacy Cucumber JSON format -- so messages.py assembles the stream. Two dependencies, each doing the half it owns: cucumber-messages, the official Python types from the protocol's own repository, for the execution messages; gherkin-official, already a transitive dependency of pytest-bdd, for the gherkinDocument and pickle payloads, which are used as it produces them rather than round-tripped through another representation. The feature files are parsed again because pytest-bdd's own dataclasses drop the AST node ids a pickle refers to. Step results come from pytest-bdd's step hooks rather than from the scenario's verdict, because a stream that marked all eight steps of a scenario failed would be saying something untrue about the seven that passed and the ones never reached. Each test case also carries a before- and after-hook TestStep: pytest runs three phases and only the middle one executes steps, so that is where a capability skip's reason and a teardown failure belong. A verdict no step accounts for -- a strict xfail that passes -- is attached to the after-hook, so it survives a consumer computing the test case's status as the worst of its steps. An expected failure is still a failure in the payload. The acknowledgement moved to the envelope's knownDeviations, declared by TckConfig.known_deviations, where it records the gap without softening the result. TckConfig also gains not_applicable, for a capability that cannot hold rather than one the provider declines. Verified locally; CI does not run on this branch, which targets the report branch rather than main. Both suites' envelopes validate against the reshaped schema with a Draft 2020-12 validator and their digests match; both streams validate clean against the Cucumber Messages JSON schema at v34.2.0 (661 messages each, zero errors). The stream accounts for all 29 collected scenarios; the five the capability gate stopped are SKIPPED for every step, none PASSED, and the one row the SDK fails is FAILED while pytest exits zero. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Go is the language that needs this first. godog counts a capability-gated skip in its passed tally, so a run that skipped five of twenty-nine scenarios prints "29 scenarios (29 passed)". Appendix F is unambiguous that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and the harness does say so in a separate log line -- but the headline number still says something false, and the number is what gets read. pytest and jest-cucumber report skips correctly, so this is the runner's property rather than the suite's design. The report does not fix godog's summary. It makes the summary stop mattering, by recording every scenario's outcome individually so a consumer can check the rule instead of trusting the runner to have applied it. The same run now reports twenty-four passed and five not-declared, each with its reason. Identity comes from revision.go, generated by sync_assets.go beside the embedded artifacts. Generating both in one command is what keeps them honest: the CI check regenerates and fails on any difference, so a revision disagreeing with the artifacts beside it cannot be committed. The check is widened to cover the generated file, which it would otherwise have missed. Both the commit and the tree hash are recorded, the tree because it identifies the artifacts alone -- unchanged by unrelated edits elsewhere in the specification, so two runs of identical artifacts agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with Config.Name recorded as the configuration, because Config.Name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read through an optional interface rather than a new BackendControl method, so that adding it breaks no existing implementation and a control that does not implement it simply omits the field. Emission is opt-in through the environment rather than through Config so that producing a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. The tests assert the property that motivated the work -- that no scenario the capability gate stopped is ever reported as passed, and that every scenario is accounted for exactly once, since a report that silently omitted what it skipped would satisfy the first half while still misleading a reader. That test earned its place immediately: it caught this emitter recording every skipped scenario twice, the second time as passed, because godog does not deliver the before hook's ErrSkip to the after hook. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

@object

A scenario entry in the conformance report was identified by feature and name. Every row of a Scenario Outline shares one name, so the type-mismatch matrix in errors.feature produced eleven entries that differed only in durationMs. If one row failed and ten passed, the report could not say which failed, and a consumer keying on feature and name kept whichever row it read last. Per the report schema, a scenarioResult now carries `example`: the Examples row it came from, keyed by column header, with the cells verbatim as strings. Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says what the table said. It is present only for outline rows and omitted otherwise. godog hands a hook an already-expanded pickle, whose step text has the parameters substituted in and whose row is otherwise gone. What survives is AstNodeIds, whose last entry is the id of the Examples TableRow. The row is therefore recovered by parsing the embedded feature files a second time and indexing every TableRow by that id. Those ids come from a counter godog creates once per run and shares across the files it parses, so reproducing them means reproducing godog's parse -- same files, same order, pickle compilation in between. That coupling is not left to be trusted: a pickle that came from an outline and did not resolve fails the run, because quietly returning to the ambiguity this field exists to remove is worse than a build failure. The capability gate records its outcome before a scenario starts, so it fills the field in too. Four skipped rows of the @object outline are as ambiguous as four failed ones. The gate's own bookkeeping is keyed by pickle id rather than by scenario name for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row and the rows that did run would have vanished from the report. gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes. Verified against the schema on open-feature/spec#425 with a Draft 2020-12 validator: every report the self-tests emit validates, the eleven matrix rows carry eleven distinct examples matching the feature file, and (feature, name, example) is unique across every scenario in a report. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Go is the language that needs this first. godog counts a capability-gated skip in its passed tally, so a run that skipped five of twenty-nine scenarios prints "29 scenarios (29 passed)". Appendix F is unambiguous that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and the harness does say so in a separate log line -- but the headline number still says something false, and the number is what gets read. pytest and jest-cucumber report skips correctly, so this is the runner's property rather than the suite's design. The report does not fix godog's summary. It makes the summary stop mattering, by recording every scenario's outcome individually so a consumer can check the rule instead of trusting the runner to have applied it. The same run now reports twenty-four passed and five not-declared, each with its reason. Identity comes from revision.go, generated by sync_assets.go beside the embedded artifacts. Generating both in one command is what keeps them honest: the CI check regenerates and fails on any difference, so a revision disagreeing with the artifacts beside it cannot be committed. The check is widened to cover the generated file, which it would otherwise have missed. Both the commit and the tree hash are recorded, the tree because it identifies the artifacts alone -- unchanged by unrelated edits elsewhere in the specification, so two runs of identical artifacts agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with Config.Name recorded as the configuration, because Config.Name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read through an optional interface rather than a new BackendControl method, so that adding it breaks no existing implementation and a control that does not implement it simply omits the field. Emission is opt-in through the environment rather than through Config so that producing a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. The tests assert the property that motivated the work -- that no scenario the capability gate stopped is ever reported as passed, and that every scenario is accounted for exactly once, since a report that silently omitted what it skipped would satisfy the first half while still misleading a reader. That test earned its place immediately: it caught this emitter recording every skipped scenario twice, the second time as passed, because godog does not deliver the before hook's ErrSkip to the after hook. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

@object

A scenario entry in the conformance report was identified by feature and name. Every row of a Scenario Outline shares one name, so the type-mismatch matrix in errors.feature produced eleven entries that differed only in durationMs. If one row failed and ten passed, the report could not say which failed, and a consumer keying on feature and name kept whichever row it read last. Per the report schema, a scenarioResult now carries `example`: the Examples row it came from, keyed by column header, with the cells verbatim as strings. Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says what the table said. It is present only for outline rows and omitted otherwise. godog hands a hook an already-expanded pickle, whose step text has the parameters substituted in and whose row is otherwise gone. What survives is AstNodeIds, whose last entry is the id of the Examples TableRow. The row is therefore recovered by parsing the embedded feature files a second time and indexing every TableRow by that id. Those ids come from a counter godog creates once per run and shares across the files it parses, so reproducing them means reproducing godog's parse -- same files, same order, pickle compilation in between. That coupling is not left to be trusted: a pickle that came from an outline and did not resolve fails the run, because quietly returning to the ambiguity this field exists to remove is worse than a build failure. The capability gate records its outcome before a scenario starts, so it fills the field in too. Four skipped rows of the @object outline are as ambiguous as four failed ones. The gate's own bookkeeping is keyed by pickle id rather than by scenario name for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row and the rows that did run would have vanished from the report. gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes. Verified against the schema on open-feature/spec#425 with a Draft 2020-12 validator: every report the self-tests emit validates, the eleven matrix rows carry eleven distinct examples matching the feature file, and (feature, name, example) is unique across every scenario in a report. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Go is the language that needs this first. godog counts a capability-gated skip in its passed tally, so a run that skipped five of twenty-nine scenarios prints "29 scenarios (29 passed)". Appendix F is unambiguous that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and the harness does say so in a separate log line -- but the headline number still says something false, and the number is what gets read. pytest and jest-cucumber report skips correctly, so this is the runner's property rather than the suite's design. The report does not fix godog's summary. It makes the summary stop mattering, by recording every scenario's outcome individually so a consumer can check the rule instead of trusting the runner to have applied it. The same run now reports twenty-four passed and five not-declared, each with its reason. Identity comes from revision.go, generated by sync_assets.go beside the embedded artifacts. Generating both in one command is what keeps them honest: the CI check regenerates and fails on any difference, so a revision disagreeing with the artifacts beside it cannot be committed. The check is widened to cover the generated file, which it would otherwise have missed. Both the commit and the tree hash are recorded, the tree because it identifies the artifacts alone -- unchanged by unrelated edits elsewhere in the specification, so two runs of identical artifacts agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with Config.Name recorded as the configuration, because Config.Name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read through an optional interface rather than a new BackendControl method, so that adding it breaks no existing implementation and a control that does not implement it simply omits the field. Emission is opt-in through the environment rather than through Config so that producing a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. The tests assert the property that motivated the work -- that no scenario the capability gate stopped is ever reported as passed, and that every scenario is accounted for exactly once, since a report that silently omitted what it skipped would satisfy the first half while still misleading a reader. That test earned its place immediately: it caught this emitter recording every skipped scenario twice, the second time as passed, because godog does not deliver the before hook's ErrSkip to the after hook. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

@object

A scenario entry in the conformance report was identified by feature and name. Every row of a Scenario Outline shares one name, so the type-mismatch matrix in errors.feature produced eleven entries that differed only in durationMs. If one row failed and ten passed, the report could not say which failed, and a consumer keying on feature and name kept whichever row it read last. Per the report schema, a scenarioResult now carries `example`: the Examples row it came from, keyed by column header, with the cells verbatim as strings. Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says what the table said. It is present only for outline rows and omitted otherwise. godog hands a hook an already-expanded pickle, whose step text has the parameters substituted in and whose row is otherwise gone. What survives is AstNodeIds, whose last entry is the id of the Examples TableRow. The row is therefore recovered by parsing the embedded feature files a second time and indexing every TableRow by that id. Those ids come from a counter godog creates once per run and shares across the files it parses, so reproducing them means reproducing godog's parse -- same files, same order, pickle compilation in between. That coupling is not left to be trusted: a pickle that came from an outline and did not resolve fails the run, because quietly returning to the ambiguity this field exists to remove is worse than a build failure. The capability gate records its outcome before a scenario starts, so it fills the field in too. Four skipped rows of the @object outline are as ambiguous as four failed ones. The gate's own bookkeeping is keyed by pickle id rather than by scenario name for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row and the rows that did run would have vanished from the report. gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes. Verified against the schema on open-feature/spec#425 with a Draft 2020-12 validator: every report the self-tests emit validates, the eleven matrix rows carry eleven distinct examples matching the feature file, and (feature, name, example) is unique across every scenario in a report. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Go is the language that needs this first. godog counts a capability-gated skip in its passed tally, so a run that skipped five of twenty-nine scenarios prints "29 scenarios (29 passed)". Appendix F is unambiguous that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and the harness does say so in a separate log line -- but the headline number still says something false, and the number is what gets read. pytest and jest-cucumber report skips correctly, so this is the runner's property rather than the suite's design. The report does not fix godog's summary. It makes the summary stop mattering, by recording every scenario's outcome individually so a consumer can check the rule instead of trusting the runner to have applied it. The same run now reports twenty-four passed and five not-declared, each with its reason. Identity comes from revision.go, generated by sync_assets.go beside the embedded artifacts. Generating both in one command is what keeps them honest: the CI check regenerates and fails on any difference, so a revision disagreeing with the artifacts beside it cannot be committed. The check is widened to cover the generated file, which it would otherwise have missed. Both the commit and the tree hash are recorded, the tree because it identifies the artifacts alone -- unchanged by unrelated edits elsewhere in the specification, so two runs of identical artifacts agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with Config.Name recorded as the configuration, because Config.Name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read through an optional interface rather than a new BackendControl method, so that adding it breaks no existing implementation and a control that does not implement it simply omits the field. Emission is opt-in through the environment rather than through Config so that producing a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. The tests assert the property that motivated the work -- that no scenario the capability gate stopped is ever reported as passed, and that every scenario is accounted for exactly once, since a report that silently omitted what it skipped would satisfy the first half while still misleading a reader. That test earned its place immediately: it caught this emitter recording every skipped scenario twice, the second time as passed, because godog does not deliver the before hook's ErrSkip to the after hook. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

@object

A scenario entry in the conformance report was identified by feature and name. Every row of a Scenario Outline shares one name, so the type-mismatch matrix in errors.feature produced eleven entries that differed only in durationMs. If one row failed and ten passed, the report could not say which failed, and a consumer keying on feature and name kept whichever row it read last. Per the report schema, a scenarioResult now carries `example`: the Examples row it came from, keyed by column header, with the cells verbatim as strings. Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says what the table said. It is present only for outline rows and omitted otherwise. godog hands a hook an already-expanded pickle, whose step text has the parameters substituted in and whose row is otherwise gone. What survives is AstNodeIds, whose last entry is the id of the Examples TableRow. The row is therefore recovered by parsing the embedded feature files a second time and indexing every TableRow by that id. Those ids come from a counter godog creates once per run and shares across the files it parses, so reproducing them means reproducing godog's parse -- same files, same order, pickle compilation in between. That coupling is not left to be trusted: a pickle that came from an outline and did not resolve fails the run, because quietly returning to the ambiguity this field exists to remove is worse than a build failure. The capability gate records its outcome before a scenario starts, so it fills the field in too. Four skipped rows of the @object outline are as ambiguous as four failed ones. The gate's own bookkeeping is keyed by pickle id rather than by scenario name for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row and the rows that did run would have vanished from the report. gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes. Verified against the schema on open-feature/spec#425 with a Draft 2020-12 validator: every report the self-tests emit validates, the eleven matrix rows carry eleven distinct examples matching the feature file, and (feature, name, example) is unique across every scenario in a report. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Unset means no report, which is the default and is not an error. An environment variable rather than a TckConfig field, so that emitting a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. Several suites in one pytest session each write their own file, so flagd's two resolvers would not collide. The load-bearing part is the per-scenario list. Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and nothing downstream can check that against a summary line. Recording every scenario's outcome individually makes the rule checkable by the consumer instead of dependent on the runner. It is also required to be complete, because a document that quietly dropped what it skipped would satisfy the letter of the rule and still mislead whoever read it. pytest, unlike godog, reports a skip honestly -- so the interesting divergence here is elsewhere. The one scenario the Python SDK cannot satisfy is marked xfail, so the run finishes green; the provider still did not satisfy it, and the document says failed with the reason. An expected failure is a recorded deviation, not an excused one. Scenarios are therefore enumerated at collection and resolved at the end of the session rather than as fixtures run, which is also what keeps a scenario skipped by a marker -- whose fixtures never run at all -- from vanishing from the document. Identity comes from spec_revision.json, generated by hatch_build_sync.py beside the copied assets and force-included into the wheel. It has to be captured at build time: the submodule that knows the answer is not in the distribution, so an installed copy has nothing left to ask. A build that cannot reach git -- an unpacked sdist -- warns and records "unknown" rather than inventing a commit. Both the commit and the tree hash are recorded, the tree because it identifies the assets alone: unchanged by unrelated edits elsewhere in the specification, so two runs of identical assets agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with TckConfig.name recorded as the configuration, because TckConfig.name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read off an optional control_api property rather than added to the BackendControl protocol, so that adding it leaves every existing control complete and one that stays quiet simply omits the field. The tests assert the two properties a consumer is entitled to assume -- that no scenario the capability gate stopped is ever reported as passed, and that every collected scenario appears exactly once, counted against pytest's own collection rather than against a number written down beside it. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

The emitter defined its own per-scenario result list: a four-value outcome enum, a tag list, a reason, and a field naming which Scenario Outline row an entry came from. All of it already exists in Cucumber Messages, which is maintained, cross-language, schema'd, and emitted natively by cucumber-jvm. The report schema was reshaped to reference a Messages payload rather than define one (open-feature/spec#425); this follows it. A run now writes two files per suite: <name>.json, the envelope, and <name>.ndjson, the results it points at, with results.digest over the exact bytes written. Deleted, because Messages carries them: scenarios[] - now TestCase/TestCaseStarted/TestStepFinished/TestCaseFinished. the outcome enum - Cucumber's own seven statuses. The declared/not-applicable distinction was never a property of the run: it follows from the declaration and the scenario's tags, so it is stated once in the envelope instead of once per scenario. example - a pickle's astNodeIds are [scenario id, table row id], and the row id resolves in the GherkinDocument to the cells the feature file wrote. Four implementations were each reinventing this field by hand. tck.assetsTree - the payload carries the executed feature Source verbatim, which answers "did two runs ask the same questions" directly rather than by proxy. Two things Messages cannot carry, so they stay. The declaration is an input to reading the results, not a summary of them. And no standard results format has a slot for the tested subject: Messages records the runtime and the OS, not what was being asked about. pytest-bdd emits no Messages -- it ships the legacy Cucumber JSON format -- so messages.py assembles the stream. Two dependencies, each doing the half it owns: cucumber-messages, the official Python types from the protocol's own repository, for the execution messages; gherkin-official, already a transitive dependency of pytest-bdd, for the gherkinDocument and pickle payloads, which are used as it produces them rather than round-tripped through another representation. The feature files are parsed again because pytest-bdd's own dataclasses drop the AST node ids a pickle refers to. Step results come from pytest-bdd's step hooks rather than from the scenario's verdict, because a stream that marked all eight steps of a scenario failed would be saying something untrue about the seven that passed and the ones never reached. Each test case also carries a before- and after-hook TestStep: pytest runs three phases and only the middle one executes steps, so that is where a capability skip's reason and a teardown failure belong. A verdict no step accounts for -- a strict xfail that passes -- is attached to the after-hook, so it survives a consumer computing the test case's status as the worst of its steps. An expected failure is still a failure in the payload. The acknowledgement moved to the envelope's knownDeviations, declared by TckConfig.known_deviations, where it records the gap without softening the result. TckConfig also gains not_applicable, for a capability that cannot hold rather than one the provider declines. Verified locally; CI does not run on this branch, which targets the report branch rather than main. Both suites' envelopes validate against the reshaped schema with a Draft 2020-12 validator and their digests match; both streams validate clean against the Cucumber Messages JSON schema at v34.2.0 (661 messages each, zero errors). The stream accounts for all 29 collected scenarios; the five the capability gate stopped are SKIPPED for every step, none PASSED, and the one row the SDK fails is FAILED while pytest exits zero. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Go is the language that needs this first. godog counts a capability-gated skip in its passed tally, so a run that skipped five of twenty-nine scenarios prints "29 scenarios (29 passed)". Appendix F is unambiguous that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and the harness does say so in a separate log line -- but the headline number still says something false, and the number is what gets read. pytest and jest-cucumber report skips correctly, so this is the runner's property rather than the suite's design. The report does not fix godog's summary. It makes the summary stop mattering, by recording every scenario's outcome individually so a consumer can check the rule instead of trusting the runner to have applied it. The same run now reports twenty-four passed and five not-declared, each with its reason. Identity comes from revision.go, generated by sync_assets.go beside the embedded artifacts. Generating both in one command is what keeps them honest: the CI check regenerates and fails on any difference, so a revision disagreeing with the artifacts beside it cannot be committed. The check is widened to cover the generated file, which it would otherwise have missed. Both the commit and the tree hash are recorded, the tree because it identifies the artifacts alone -- unchanged by unrelated edits elsewhere in the specification, so two runs of identical artifacts agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with Config.Name recorded as the configuration, because Config.Name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read through an optional interface rather than a new BackendControl method, so that adding it breaks no existing implementation and a control that does not implement it simply omits the field. Emission is opt-in through the environment rather than through Config so that producing a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. The tests assert the property that motivated the work -- that no scenario the capability gate stopped is ever reported as passed, and that every scenario is accounted for exactly once, since a report that silently omitted what it skipped would satisfy the first half while still misleading a reader. That test earned its place immediately: it caught this emitter recording every skipped scenario twice, the second time as passed, because godog does not deliver the before hook's ErrSkip to the after hook. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

@object

A scenario entry in the conformance report was identified by feature and name. Every row of a Scenario Outline shares one name, so the type-mismatch matrix in errors.feature produced eleven entries that differed only in durationMs. If one row failed and ten passed, the report could not say which failed, and a consumer keying on feature and name kept whichever row it read last. Per the report schema, a scenarioResult now carries `example`: the Examples row it came from, keyed by column header, with the cells verbatim as strings. Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says what the table said. It is present only for outline rows and omitted otherwise. godog hands a hook an already-expanded pickle, whose step text has the parameters substituted in and whose row is otherwise gone. What survives is AstNodeIds, whose last entry is the id of the Examples TableRow. The row is therefore recovered by parsing the embedded feature files a second time and indexing every TableRow by that id. Those ids come from a counter godog creates once per run and shares across the files it parses, so reproducing them means reproducing godog's parse -- same files, same order, pickle compilation in between. That coupling is not left to be trusted: a pickle that came from an outline and did not resolve fails the run, because quietly returning to the ambiguity this field exists to remove is worse than a build failure. The capability gate records its outcome before a scenario starts, so it fills the field in too. Four skipped rows of the @object outline are as ambiguous as four failed ones. The gate's own bookkeeping is keyed by pickle id rather than by scenario name for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row and the rows that did run would have vanished from the report. gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes. Verified against the schema on open-feature/spec#425 with a Draft 2020-12 validator: every report the self-tests emit validates, the eleven matrix rows carry eleven distinct examples matching the feature file, and (feature, name, example) is unique across every scenario in a report. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to <dir>/<name>.json against the report schema in the specification repository (open-feature/spec#425, part of open-feature/spec#424). Unset means no report, which is the default and is not an error. An environment variable rather than a TckConfig field, so that emitting a report is a property of the run and not of the code: CI sets it, a local run does not, and no adopter changes a line to publish one. Several suites in one pytest session each write their own file, so flagd's two resolvers would not collide. The load-bearing part is the per-scenario list. Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and nothing downstream can check that against a summary line. Recording every scenario's outcome individually makes the rule checkable by the consumer instead of dependent on the runner. It is also required to be complete, because a document that quietly dropped what it skipped would satisfy the letter of the rule and still mislead whoever read it. pytest, unlike godog, reports a skip honestly -- so the interesting divergence here is elsewhere. The one scenario the Python SDK cannot satisfy is marked xfail, so the run finishes green; the provider still did not satisfy it, and the document says failed with the reason. An expected failure is a recorded deviation, not an excused one. Scenarios are therefore enumerated at collection and resolved at the end of the session rather than as fixtures run, which is also what keeps a scenario skipped by a marker -- whose fixtures never run at all -- from vanishing from the document. Identity comes from spec_revision.json, generated by hatch_build_sync.py beside the copied assets and force-included into the wheel. It has to be captured at build time: the submodule that knows the answer is not in the distribution, so an installed copy has nothing left to ask. A build that cannot reach git -- an unpacked sdist -- warns and records "unknown" rather than inventing a commit. Both the commit and the tree hash are recorded, the tree because it identifies the assets alone: unchanged by unrelated edits elsewhere in the specification, so two runs of identical assets agree even when pinned to different commits, and checkable because `git rev-parse <commit>:specification/assets/provider-tck` reproduces it. Two smaller decisions. The provider is identified by the name it reports through its own metadata, with TckConfig.name recorded as the configuration, because TckConfig.name is chosen to read well in a failure message -- "flagd-rpc" -- and a provider with two materially different modes produces two reports that are not interchangeable. And how the backend was driven is read off an optional control_api property rather than added to the BackendControl protocol, so that adding it leaves every existing control complete and one that stays quiet simply omits the field. The tests assert the two properties a consumer is entitled to assume -- that no scenario the capability gate stopped is ever reported as passed, and that every collected scenario appears exactly once, counted against pytest's own collection rather than against a number written down beside it. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

The emitter defined its own per-scenario result list: a four-value outcome enum, a tag list, a reason, and a field naming which Scenario Outline row an entry came from. All of it already exists in Cucumber Messages, which is maintained, cross-language, schema'd, and emitted natively by cucumber-jvm. The report schema was reshaped to reference a Messages payload rather than define one (open-feature/spec#425); this follows it. A run now writes two files per suite: <name>.json, the envelope, and <name>.ndjson, the results it points at, with results.digest over the exact bytes written. Deleted, because Messages carries them: scenarios[] - now TestCase/TestCaseStarted/TestStepFinished/TestCaseFinished. the outcome enum - Cucumber's own seven statuses. The declared/not-applicable distinction was never a property of the run: it follows from the declaration and the scenario's tags, so it is stated once in the envelope instead of once per scenario. example - a pickle's astNodeIds are [scenario id, table row id], and the row id resolves in the GherkinDocument to the cells the feature file wrote. Four implementations were each reinventing this field by hand. tck.assetsTree - the payload carries the executed feature Source verbatim, which answers "did two runs ask the same questions" directly rather than by proxy. Two things Messages cannot carry, so they stay. The declaration is an input to reading the results, not a summary of them. And no standard results format has a slot for the tested subject: Messages records the runtime and the OS, not what was being asked about. pytest-bdd emits no Messages -- it ships the legacy Cucumber JSON format -- so messages.py assembles the stream. Two dependencies, each doing the half it owns: cucumber-messages, the official Python types from the protocol's own repository, for the execution messages; gherkin-official, already a transitive dependency of pytest-bdd, for the gherkinDocument and pickle payloads, which are used as it produces them rather than round-tripped through another representation. The feature files are parsed again because pytest-bdd's own dataclasses drop the AST node ids a pickle refers to. Step results come from pytest-bdd's step hooks rather than from the scenario's verdict, because a stream that marked all eight steps of a scenario failed would be saying something untrue about the seven that passed and the ones never reached. Each test case also carries a before- and after-hook TestStep: pytest runs three phases and only the middle one executes steps, so that is where a capability skip's reason and a teardown failure belong. A verdict no step accounts for -- a strict xfail that passes -- is attached to the after-hook, so it survives a consumer computing the test case's status as the worst of its steps. An expected failure is still a failure in the payload. The acknowledgement moved to the envelope's knownDeviations, declared by TckConfig.known_deviations, where it records the gap without softening the result. TckConfig also gains not_applicable, for a capability that cannot hold rather than one the provider declines. Verified locally; CI does not run on this branch, which targets the report branch rather than main. Both suites' envelopes validate against the reshaped schema with a Draft 2020-12 validator and their digests match; both streams validate clean against the Cucumber Messages JSON schema at v34.2.0 (661 messages each, zero errors). The stream accounts for all 29 collected scenarios; the five the capability gate stopped are SKIPPED for every step, none PASSED, and the one row the SDK fails is FAILED while pytest exits zero. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

First step of the design discussed in #424: the report format, before any implementation emits one, so four languages do not invent four dialects. Three properties are deliberate and are the parts worth arguing about. Four outcomes, not two. not-declared and not-applicable are distinct, and collapsing them misrepresents a provider: @strict-numeric-typing is unsatisfiable in JavaScript because the language has no integer type, so reporting it as not-declared would show every JS provider as missing something no JS provider can have. Per-scenario results are required, and required to be complete. That is what makes the appendix's own rule -- a skipped scenario is never reported as passed -- checkable rather than aspirational. It is not hypothetical: godog's summary counts capability-gated skips as passes, so the Go runner satisfies the rule only in a separate log line. A runner with that flaw still produces a truthful report here. The spec revision sits inside the tck object rather than standing alone, because it is a property of the artifact that ran rather than an independent assertion, and it travels with a digest of the assets actually executed. The revision is a claim; the digest is the check that catches a locally edited vendored file. There is deliberately no aggregate score. A single percentage would reward declaring fewer capabilities, which inverts the incentive the suite exists to create. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

… digest The schema carried a `gherkinDigest`, a SHA-256 over the feature files and the canonical flag set, on the argument that `specRevision` is an assertion and a digest is a check. The check is worth having; computing it ourselves is not. It would have made every implementation carry a normative hashing algorithm -- agreeing on file selection, path form, ordering, framing and line-ending normalisation, the last of which matters because Go embeds committed copies that a Windows checkout converts to CRLF. Four implementations of that is a lot of surface for something git already computes. `git rev-parse <revision>:specification/assets/provider-tck` is the same idea with none of the specification. It has two properties the digest was reaching for. It is verifiable: a third party can recompute it from the recorded revision, so a revision recorded wrongly does not pass unnoticed. And it is stable across unrelated spec commits, so two runs that executed identical assets report the same value even when pinned to different commits -- which the commit SHA alone gets wrong, since it changes for every unrelated edit to the specification. The drift the digest was chiefly aimed at -- a vendored copy edited locally while the recorded revision still points at the original -- is in any case already handled where it arises. Java, Python and JavaScript read the assets straight from the submodule, so no second copy exists to diverge. Go has committed copies and a CI check that regenerates them and fails on any difference. It is a build-time problem with a build-time fix, and pushing it into the report format would have every implementation re-detect what its own CI already catches. The cost is that the tree covers the asset directory's documentation too, so a README edit changes it. That errs towards reporting two comparable runs as different rather than the reverse, which is the safe direction. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Three changes, all of them found by implementing the format rather than by reading it. Writing four emitters against this schema was always going to be the real review, and it was. A reason is now required whenever an outcome is not "passed", at both the capability and the scenario level. The schema previously said it was "required in practice", which is another way of saying not required: nothing enforced it, and a report of bare tags with no explanation would have validated. The reason is most of the value to a person comparing providers -- "does not support configuration-change events" is information, "@configuration-change: not-declared" is a shrug. knownDeviation moves to a shared definition and becomes available on a scenario as well as a capability, because the two are different claims. A capability-level deviation says the provider does not do X. A scenario-level one pins the deviation to the single case that fails, which is what an implementation marking one expected failure has to record -- the Python suite marks exactly one, for open-feature/python-sdk#619, and until now had nowhere to put it but prose in the reason string. The capabilities object now states plainly that it is not a verdict. Scenarios carrying no capability tag are mandatory, roll up into no capability, and are therefore invisible in that summary -- so a provider can fail a mandatory scenario while every entry reads "passed". The Python implementation produced exactly that report. A consumer deciding whether a provider conforms has to read the scenarios, which is the reason those are required and required to be complete, and the schema should say so where someone will read it rather than leaving it to be discovered. All reports emitted by the Go, Python and JavaScript implementations still validate, and the two new constraints were checked to reject a report that omits a reason rather than merely being decorative. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

A scenario entry was identified by feature and name, and every row of a Scenario Outline shares one name. The type-mismatch matrix is eleven rows, so a report carried eleven entries distinguished by nothing but a duration measurement. When one row fails and ten pass -- which is what the Python implementation actually produced -- the report cannot say which failed, and a consumer keying on feature and name keeps whichever row it happened to see last. An entry now carries the Examples row it came from, as the row's parameters keyed by column header, and identity is feature, name and example together. The parameters are the identity, and they come from the feature file rather than from any runner, which is why this is a field rather than a naming convention. Mandating a mangled name would have put a separator, an ordering and an escaping rule for values containing the separator into normative text that four implementations must reproduce byte-for-byte, where drift stays invisible until two reports quietly fail to line up. That was not a hypothetical risk. Before this field existed the three implementations had already diverged on exactly this point, each reasonably and none compatibly: Go emitted the bare scenario name for all eleven rows, Python appended pytest's example id, JavaScript used jest-cucumber's expanded title. The same row of the same feature file had three identities, which defeats the comparison the format exists to make possible. Values are the cell contents verbatim as strings, because Gherkin has no types. Coercing "1" to a number here would make the report disagree with the table it came from. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Two implementations asked independently whether the four languages should agree on the key order within an example object. They currently do not: Java emits Examples column order, Go sorts alphabetically because encoding/json sorts map keys when marshalling, and neither is wrong. A JSON object is unordered, so the answer is that order carries no meaning and implementations need not agree. Saying so explicitly is worth a sentence, because the alternative is three more implementations each deciding privately and someone eventually byte-comparing two reports and concluding they disagree when they do not. If byte-level reproducibility is ever required -- an attestation over the document is the obvious case -- that is a canonicalisation problem for the whole report, not something to solve inside one field. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

The schema defined its own per-scenario result list, a four-value outcome enum, and a field identifying which Scenario Outline row an entry came from. All three already exist in Cucumber Messages, which is maintained, cross-language and schema'd, and which cucumber-jvm emits natively. The example field is the clearest case for dropping ours. It was added days ago to make outline rows identifiable, and four implementations each reinvented it independently -- one of them by reverse-engineering how its runner maps a pickle back to a table row. Messages carries that identity as AST node ids and has all along. Messages is also better than what it replaces. It carries the executed feature source, so a consumer can verify which questions were asked rather than trusting a recorded revision, which is stronger than the tree hash this schema had been using for the same purpose. Two things Messages cannot carry, so they stay here. The declaration -- which capabilities the provider claims -- is an input to reading the results rather than a summary of them. A skipped scenario says the question was not put to this provider; only the declaration says whether that is because the provider declines the capability. Given the declaration and a scenario's tags, the reason for a skip follows, so it no longer needs transporting per scenario. The tested subject and its provenance. No standard results format has a slot for "the provider under test": Messages records the runtime and OS, not the subject. The results are referenced rather than inlined, because a Messages stream carries the feature sources and is far larger than this envelope, and a consumer deciding whether it cares about a report should not have to fetch a whole run to find out. Two lossier formats are admitted for runners that cannot emit Messages yet, with what each loses stated. Not addressed here, and still open on #424: whether a report should be published inside the provider's released artifact, and how strongly the tested subject revision must be identified. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Two corrections, both found by implementing this in Go. assetsTree is gone. It identified the executed artifacts so that a recorded spec revision could be checked rather than trusted. Cucumber Messages carries the executed feature source itself, so the check is now the source, and keeping a hash of the same thing alongside it is a second place for the answer to be wrong. The previous commit dropped it from the emitters and neglected to drop it here. formatVersion is new, and its absence was a real hole. Cucumber Messages is versioned, and implementations pin different releases: the Go TCK builds against messages v21, while cucumber-jvm ships a considerably later one and the npm package is later still. The message types differ between releases. Without recording what the producer built against, a consumer validating a stream has to guess which schema to validate against -- and guessing wrong is worse than not validating at all, because a later schema accepts messages an earlier producer could not have emitted, while an earlier one rejects messages that are perfectly valid. That hazard is not hypothetical. During this work a schema copy of the wrong version was left in a shared temporary directory, and validating a v21 stream against it would have reported success or failure for reasons unrelated to the stream. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

… stream Two corrections from implementing this in Java. The claim that a Messages stream makes the asset tree hash redundant was wrong. The stream carries the executed feature source, so the features are corroborated rather than asserted -- but it says nothing about the canonical flag set or the control API description, and a scenario asserting that integer-flag resolves to 10 depends entirely on the flag set it ran against. specRevision now carries those two alone, and the schema says so instead of implying the stream covers everything. The tree hash stays dropped, on the same reasoning as before: every implementation either reads the assets from the submodule or regenerates and diffs its vendored copy in CI, so drift is caught where it happens. Second, a testCaseFinished message carries no status. A scenario's outcome is the most severe testStepResult among its steps, hooks included. That ordering is what makes a capability skip visible at all, since a gate aborting in a before-hook produces a SKIPPED hook result that outranks the PASSED steps it prevented -- and a consumer reading only testCaseFinished sees no outcome whatsoever. Adopting a standard format means inheriting its sharp edges, and this is the one most likely to mislead someone reading a report for the first time. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

@targeting

The declaration replaced the per-capability rollup this schema used to carry, and inherited none of its discipline. The rollup omitted a capability no scenario exercised, so an untested capability never reached a report; the declaration lists whatever the provider claims, so a real report now asserts @targeting and @caching -- reserved tags with no scenarios behind them. That is the third route to one vacuous claim. It was closed once when the rollup reported such a capability as passed, once when a capability whose every scenario had been skipped still read passed, and now again through a field that looked like a plain transcription of the provider's own configuration. So the field now says what belongs in it: capabilities the provider declares and the executed suite gates on. A capability nothing tests cannot produce a skip, so it plays no part in interpreting the results, and listing it invites a reader to believe it was verified. It also says where the complaint goes. An implementation finding a declared capability the suite does not test should warn or fail, so the adopter hears about it -- silence in the report and silence to the adopter are different things, and this got in because an adopter declaring "everything except X" picks up every reserved tag on the way past. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

The field's description illustrates a not-applicable capability with @strict-numeric-typing, which was renamed to @numeric-coercion. Prose only, so nothing failed validation -- which is exactly why it survived the rename: a tag inside a description is invisible to every check that would otherwise have caught it. Found by an implementation updating its own references and noticing the schema it validates against still used the old name. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

"Not declared" and "not applicable" are both skips. Giving them separate representations asks every implementation to agree on more vocabulary without telling a reader anything the skip's reason does not already say: the scenario's tags say what was asked, `declared` says whether it was claimed, and the reason says why it was skipped. The field's own description made the argument for removing it. It reserved itself for provider-specific impossibility, on the grounds that "where the impossibility is a property of the language rather than of the provider it is better recorded once in the capability documentation than repeated in every report" -- and both motivating cases are exactly that. @numeric-coercion cannot hold where the language has a single numeric type; @large-integers cannot hold on a 32-bit accessor. Neither is a fact about a provider, and both are now stated in Appendix F. Four implementations built the field. No adoption in any of them populates it, so the provider-specific case it was kept for has not arisen in four languages. The declaration's description now says where language-level impossibility lives, so a reader who looks for this field finds the answer rather than its absence. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

The schema landed without the appendix ever describing the document it describes. Appendix F named reports twenty-three times, defined none of them, and said nothing at all about provenance -- which is the shape of gap the appendix exists to close, since a field four implementations can read two ways is one they will implement two ways. Provenance is optional and nobody emits it, and both of those are now stated rather than left to be inferred from silence. It is optional because its contents are properties of the environment that ran the suite: outside CI there is no run URL, and a working tree with uncommitted changes has no commit that describes what was executed. A report that filled the field in regardless would be worse than one that omits it, because the value of the field is precisely that a consumer can check it. That is also the argument for keeping it where the same argument removed declaration.notApplicable. That field was a second way to say what a skip's reason already said. This one carries facts nothing else in the document carries, and an absent optional object is unambiguous in a way a missing not-applicable entry was not. The section also writes down why declaration.declared and knownDeviations sit where they do, because both are read alongside the results and neither can be derived from them. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

…andatory The knownDeviation definition had no required array, so a deviation carrying neither a summary nor an issue validated. summary is now required and non-empty: one that does not say what the gap is leaves a reader worse off than the bare skip or failure it accompanies, which is the thing it exists to explain. issue stays optional and now says so. The description claimed the gap was "tracked somewhere" while the schema required nothing of the kind, and one implementation read that as normative -- Python made issue a mandatory field with the docstring "a URI, because the schema requires one", leaving it the only one of four that cannot record an untracked defect. Naming an untracked defect is precisely what distinguishes it from a capability the provider chose to withhold, so the schema should not have been read as forbidding it and now cannot be. Appendix F carries which of the two legitimate shapes to prefer; the description points at it rather than restating it. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

…report is worth The field said how the backend was driven and nothing required it, so four implementations arrived at four answers: a default method returning "in-process" in Java, an optional interface field in JavaScript, a comment-documented duck-typed property in Python, and in Go an unexported optional interface on the report side with a doc comment conceding that "silence here is a small lie by omission". Each was locally reasonable and together they mean a consumer cannot rely on the field being there. It is not a detail. The same scenarios passing over the normative control API and passing through in-process manipulation of a provider that has a real backend are not the same claim, and this is the only field that separates them. So it is required, and required to come from the control: nothing outside the control can tell which path a run used, and a harness that infers it from the control's concrete type is right about its own built-ins and quietly wrong about an adopter's custom control -- the one case where the answer matters. backend becomes required with it. It was documented as "omitted for a provider with no backend", which contradicted the controlApi enum, whose "in-process" member exists for exactly that provider: the one value most worth knowing could never legally appear. Every run manipulated flag state somehow, so every report can say how. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

The declaration's description claimed that "given the declaration and a scenario's tags, the reason for a skip follows without needing to be transported per scenario", and then two sentences later conceded that an inexpressible capability's skip "carries the reason". Both cannot be true, and the flat claim came first, so a consumer would build on it. Appendix F now requires an implementation to refuse a capability its SDK cannot express rather than leaving adopters to withhold it, which makes the gap permanent and uniform: @large-integers is absent from every Java declaration and @numeric-coercion from every JavaScript one, in both cases for a reason that says nothing about the provider. A reader who infers "the provider declined" from an absent capability is now wrong in the direction that matters -- it reads as a deliberate limitation of a specific provider when it is a property of the language. So the description says what is actually derivable and what is not: most skips follow from the declaration, the inexpressible ones do not and cannot, Appendix F records which capabilities that applies to per language, and the results payload carries the distinction per skip. Found by the Java implementation while moving @large-integers into the new refusal. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

aepfli mentioned this pull request Aug 24, 2026

[Tracking] A cross-language conformance suite for OpenFeature providers #417

Open

aepfli force-pushed the feat/provider-tck-report-schema branch from 5adef5b to 56ed4f9 Compare August 24, 2026 19:57

aepfli force-pushed the feat/provider-tck-report-schema branch 5 times, most recently from 4b50e46 to 97dc46a Compare September 11, 2026 07:23

aepfli force-pushed the feat/provider-tck-appendix branch from 4c88e46 to 2acd860 Compare September 11, 2026 07:27

aepfli force-pushed the feat/provider-tck-report-schema branch from 97dc46a to 71c7ad7 Compare September 11, 2026 07:27

aepfli force-pushed the feat/provider-tck-report-schema branch from 2fa9bfd to 62678d9 Compare September 11, 2026 09:45

aepfli mentioned this pull request Sep 11, 2026

feat: add Appendix F, provider conformance (TCK) #423

Draft

aepfli force-pushed the feat/provider-tck-report-schema branch from 62678d9 to ba0c4ab Compare September 11, 2026 09:47

aepfli force-pushed the feat/provider-tck-report-schema branch from e141d9f to 9b3e395 Compare September 11, 2026 21:18

aepfli force-pushed the feat/provider-tck-report-schema branch 2 times, most recently from f965f73 to e14cf45 Compare September 12, 2026 10:02

aepfli added 14 commits September 12, 2026 13:04

Navigation Menu

Uh oh!

feat: add a schema for machine-readable provider conformance reports - #425

feat: add a schema for machine-readable provider conformance reports#425
aepfli wants to merge 15 commits into
feat/provider-tck-appendixfrom
feat/provider-tck-report-schema

aepfli commented Aug 24, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 24, 2026 •

edited

Loading

Draft PR not reviewed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

aepfli commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed since the first draft

What the envelope still has to carry

Results are referenced, not inlined

What is not here

Uh oh!

coderabbitai Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Draft PR not reviewed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aepfli commented Aug 24, 2026 •

edited

Loading

coderabbitai Bot commented Aug 24, 2026 •

edited

Loading