feat(provider-tck): emit a machine-readable conformance report

Setting PROVIDER_TCK_REPORT_DIR makes each suite write its run to
<dir>/<name>.json against the report schema in the specification repository
(open-feature/spec#425, part of open-feature/spec#424). Unset means no report,
which is the default and is not an error.

An environment variable rather than a TckConfig field, so that emitting a report
is a property of the run and not of the code: CI sets it, a local run does not,
and no adopter changes a line to publish one. Several suites in one pytest
session each write their own file, so flagd's two resolvers would not collide.

The load-bearing part is the per-scenario list. Appendix F requires that a
scenario skipped for an undeclared capability is reported as skipped with the
reason and never as passed, and nothing downstream can check that against a
summary line. Recording every scenario's outcome individually makes the rule
checkable by the consumer instead of dependent on the runner. It is also
required to be complete, because a document that quietly dropped what it skipped
would satisfy the letter of the rule and still mislead whoever read it.

pytest, unlike godog, reports a skip honestly -- so the interesting divergence
here is elsewhere. The one scenario the Python SDK cannot satisfy is marked
xfail, so the run finishes green; the provider still did not satisfy it, and the
document says failed with the reason. An expected failure is a recorded
deviation, not an excused one. Scenarios are therefore enumerated at collection
and resolved at the end of the session rather than as fixtures run, which is
also what keeps a scenario skipped by a marker -- whose fixtures never run at
all -- from vanishing from the document.

Identity comes from spec_revision.json, generated by hatch_build_sync.py beside
the copied assets and force-included into the wheel. It has to be captured at
build time: the submodule that knows the answer is not in the distribution, so
an installed copy has nothing left to ask. A build that cannot reach git -- an
unpacked sdist -- warns and records "unknown" rather than inventing a commit.
Both the commit and the tree hash are recorded, the tree because it identifies
the assets alone: unchanged by unrelated edits elsewhere in the specification,
so two runs of identical assets agree even when pinned to different commits, and
checkable because `git rev-parse <commit>:specification/assets/provider-tck`
reproduces it.

Two smaller decisions. The provider is identified by the name it reports through
its own metadata, with TckConfig.name recorded as the configuration, because
TckConfig.name is chosen to read well in a failure message -- "flagd-rpc" -- and
a provider with two materially different modes produces two reports that are not
interchangeable. And how the backend was driven is read off an optional
control_api property rather than added to the BackendControl protocol, so that
adding it leaves every existing control complete and one that stays quiet simply
omits the field.

The tests assert the two properties a consumer is entitled to assume -- that no
scenario the capability gate stopped is ever reported as passed, and that every
collected scenario appears exactly once, counted against pytest's own collection
rather than against a number written down beside it.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Author: aepfli

tools/openfeature-provider-tck/.gitignore

@@ -5,3 +5,6 @@
 src/openfeature/contrib/tools/provider_tck/gherkin/
 src/openfeature/contrib/tools/provider_tck/flag_data/
 src/openfeature/contrib/tools/provider_tck/control-api.yaml
+# Generated alongside them, from the submodule pin, so a conformance report can
+# name the spec revision it ran against.
+src/openfeature/contrib/tools/provider_tck/spec_revision.json

tools/openfeature-provider-tck/README.md

@@ -478,6 +478,72 @@ is recorded by the pin and nowhere else, so the two cannot drift apart unnoticed
 
 This mirrors what `openfeature-flagd-api-testkit` already does for the flagd test harness.
 
+## Conformance reports
+
+Set `PROVIDER_TCK_REPORT_DIR` and each suite writes a machine-readable record of its run to
+`<dir>/<name>.json`, conforming to the [report schema][report-schema] in the specification.
+
+```console
+$ PROVIDER_TCK_REPORT_DIR=./reports pytest
+provider-tck [in-memory]: report written to reports/in-memory.json (1 failed, 5 not-declared, 23 passed)
+
+$ jq '.scenarios | group_by(.outcome) | map({(.[0].outcome): length}) | add' reports/in-memory.json
+{
+  "failed": 1,
+  "not-declared": 5,
+  "passed": 23
+}
+```
+
+It is an environment variable rather than a `TckConfig` field so that emitting a report is a property
+of the *run* and not of the code: CI sets it, a developer running the suite locally does not, and no
+adopter changes a line to publish one. Unset means no report, which is not an error. Several suites
+in one pytest session each write their own file, so flagd's two resolvers would not collide.
+
+### Why every scenario is listed
+
+Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped
+**with the reason** and never as passed. A consumer cannot check that against a summary line, so the
+report records the outcome of *every* scenario individually — and is required to be complete, because
+a document that quietly dropped what it skipped would satisfy the letter of the rule and still
+mislead whoever read it.
+
+Which also means the report is not a transcription of pytest's summary. The run above finishes green:
+the one scenario the Python SDK cannot satisfy is marked `xfail` (finding 1), so pytest counts it as
+expected and exits zero. The provider still did not satisfy it, and the document says `failed` with
+the reason — an expected failure is a recorded deviation, not an excused one.
+
+Four outcomes rather than two, because "did not run" is not one thing:
+
+| Outcome | Means |
+| --- | --- |
+| `passed` | the scenario ran and passed |
+| `failed` | the scenario ran and failed, including a known deviation marked `xfail` |
+| `not-declared` | skipped because the provider did not declare a capability the scenario is tagged with |
+| `not-applicable` | skipped for any other reason — a marker an adopter applied, a step calling `pytest.skip` |
+
+### What identifies a report
+
+`tck.specRevision` and `tck.assetsTree` come from `spec_revision.json`, which `hatch_build_sync.py`
+generates from the submodule alongside the copied assets. It has to be captured at build time: the
+submodule is not in the wheel, so an installed copy has nothing left to ask. A build that cannot
+reach git — an unpacked sdist, say — warns and records `unknown` rather than inventing a commit.
+
+The tree hash is carried as well as the commit because it identifies the assets alone. It is
+unchanged by unrelated edits elsewhere in the specification, so two runs that executed identical
+assets report the same value even when pinned to different commits — and it is checkable, since
+`git rev-parse <specRevision>:specification/assets/provider-tck` must reproduce it.
+
+`provider.name` is what the provider reports through its own metadata, not `TckConfig.name`.
+`TckConfig.name` is chosen to read well in a failure message — `flagd-rpc` — which makes it the
+*configuration*, and it is reported as such. One provider with two materially different modes
+produces two reports that are not interchangeable.
+
+`backend.controlApi` is read off an optional `control_api` property on your `BackendControl`,
+returning `"http"` or `"in-process"`. It is not a member of the protocol: adding one would make every
+existing control incomplete for the sake of one string, and a control that stays quiet simply omits
+the field.
+
 ## The self-tests
 
 | Suite | Subject | Why |
@@ -489,14 +555,15 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes
 | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it |
 | `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity |
 | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping and the control-API it reports, against a stubbed control API |
+| `test_report` | the conformance report | checks the two properties a consumer is entitled to assume |
 
 ```
-163 passed, 35 skipped, 2 xfailed
+PLACEHOLDER passed, 27 skipped, 2 xfailed
 ```
 
-No Docker and no network beyond loopback. The conformance suites take under a second;
-`test_extensions` takes most of the rest, because the properties it checks are properties of a whole
-pytest session and it runs a generated adoption in a subprocess to check them.
+No Docker and no network beyond loopback. The conformance suites take under a second; `test_report`
+and `test_extensions` take most of the rest, because the properties they check are properties of a
+whole pytest session and they run generated adoptions in subprocesses to check them.
 
 Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios — three about
 initialisation, three about shutdown — are skipped in both. That is the point: with no backend to
@@ -521,7 +588,14 @@ both. Both declare `@variants`, since an in-memory flag set is keyed by variant
   stack and discovering its mapped ports is still each adopter's own code. Abstracting that from a
   single example tends to produce the wrong abstraction; it should wait for a second adopter.
 - **Caching, hooks and flag metadata** are not covered.
-
+- **A report cannot name a Scenario Outline row portably.** Every row of an outline shares one
+  scenario name, and the report schema has nowhere to put the row, so several entries would be
+  indistinguishable — including, here, one that differs in outcome from its siblings. This
+  implementation qualifies the name with pytest's example id (`... [boolean-flag-Integer-1]`), which
+  is unambiguous but is not what another language would produce for the same row. Raised on
+  [open-feature/spec#424](https://github.com/open-feature/spec/issues/424).
+
+[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json
 [appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md
 [appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md
 [spec]: https://github.com/open-feature/spec

tools/openfeature-provider-tck/hatch_build.py

@@ -18,15 +18,28 @@
 # the single definition of what gets copied where -- would not be importable.
 sys.path.insert(0, str(Path(__file__).parent))
 
-from hatch_build_sync import FILES, PACKAGE_REL, SPEC_ASSETS, TREES, sync
+from hatch_build_sync import (
+    FILES,
+    PACKAGE_REL,
+    REVISION_FILE,
+    SPEC_ASSETS,
+    TREES,
+    sync,
+)
 
 
 class SpecAssetsCopyHook(BuildHookInterface):
     PLUGIN_NAME = "spec-assets-copy"
 
     def initialize(self, version: str, build_data: dict) -> None:
         root = Path(self.root)
-        copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES]
+        # The generated revision file travels with the assets it describes. It
+        # has to be built here rather than read at run time, because the
+        # submodule that knows the answer is not in the wheel and a conformance
+        # report has to name the revision it ran against.
+        copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES] + [
+            root / PACKAGE_REL / REVISION_FILE
+        ]
 
         # Building from a checkout: refresh from the submodule, so what ships is
         # always the revision the pin names. Building from an sdist: there is no

tools/openfeature-provider-tck/hatch_build_sync.py

@@ -10,11 +10,16 @@
 needs no submodule: the copies are inside the distribution.
 """
 
+import json
 import shutil
+import subprocess
+import warnings
 from pathlib import Path
 
 ROOT = Path(__file__).parent
-SPEC_ASSETS = (ROOT / "spec/specification/assets/provider-tck").resolve()
+SPEC_ROOT = (ROOT / "spec").resolve()
+ASSETS_PATH_IN_SPEC = "specification/assets/provider-tck"
+SPEC_ASSETS = (SPEC_ROOT / ASSETS_PATH_IN_SPEC).resolve()
 PACKAGE_REL = Path("src/openfeature/contrib/tools/provider_tck")
 DEST_BASE = ROOT / PACKAGE_REL
 
@@ -35,6 +40,27 @@
 TREES = [("gherkin", "gherkin"), ("flags", "flag_data")]
 FILES = [("openapi/control-api.yaml", "control-api.yaml")]
 
+REVISION_FILE = "spec_revision.json"
+"""Which revision of the specification the copied assets came from.
+
+Recorded at build time because the answer is only available at build time: the
+submodule that holds it is not in the wheel, and a conformance report that cannot
+name the revision it ran against cannot be compared with another. It is generated
+by the same command that copies the assets, which is what keeps the two from
+disagreeing.
+
+Not committed, for the same reason the assets are not: the submodule pin is the
+single record of which revision this package targets.
+"""
+
+UNKNOWN_REVISION = "unknown"
+"""Seven characters, the minimum the report schema accepts.
+
+A build that cannot reach git says it does not know rather than inventing a
+commit, and still produces a document that validates. Which happens for real:
+building from a source tarball has no ``.git`` to ask.
+"""
+
 
 def sync() -> None:
     if not SPEC_ASSETS.exists():
@@ -58,6 +84,50 @@ def sync() -> None:
             dest.unlink()
         shutil.copy2(SPEC_ASSETS / src_name, dest)
 
+    write_revision()
+
+
+def write_revision() -> None:
+    """Record the spec commit and the asset tree these copies came from.
+
+    The tree hash is carried as well as the commit because it identifies the
+    assets alone: it does not change when an unrelated part of the specification
+    does, so two runs that executed identical assets report the same value even
+    when pinned to different commits. It is also checkable rather than merely
+    asserted, since ``git rev-parse <commit>:specification/assets/provider-tck``
+    must reproduce it.
+    """
+    commit = _git("rev-parse", "HEAD") or UNKNOWN_REVISION
+    tree = _git("rev-parse", f"HEAD:{ASSETS_PATH_IN_SPEC}") or ""
+    (DEST_BASE / REVISION_FILE).write_text(
+        json.dumps({"specRevision": commit, "assetsTree": tree}, indent=2) + "\n",
+        encoding="utf-8",
+    )
+
+
+def _git(*args: str) -> str:
+    """Run git inside the submodule, returning its output or an empty string.
+
+    A build must not hard-fail because git is absent or the checkout is not a
+    repository -- both are ordinary when building from an unpacked sdist. The
+    failure is reported as a warning and the identity degrades to ``unknown``,
+    which is legible in the resulting report rather than silently wrong.
+    """
+    command = ["git", "-C", str(SPEC_ROOT), *args]
+    try:
+        completed = subprocess.run(  # noqa: S603
+            command, capture_output=True, check=True, text=True
+        )
+    except (OSError, subprocess.CalledProcessError) as error:
+        warnings.warn(
+            f"could not determine the spec revision ({' '.join(command)}: {error}); "
+            f"conformance reports from this build will not name the revision they "
+            f"ran against",
+            stacklevel=2,
+        )
+        return ""
+    return completed.stdout.strip()
+
 
 if __name__ == "__main__":
     sync()

tools/openfeature-provider-tck/pyproject.toml

@@ -58,6 +58,10 @@ artifacts = [
     "src/openfeature/contrib/tools/provider_tck/gherkin/",
     "src/openfeature/contrib/tools/provider_tck/flag_data/",
     "src/openfeature/contrib/tools/provider_tck/control-api.yaml",
+    # Which spec revision those assets came from, generated beside them. The
+    # submodule is not in the wheel, so a conformance report emitted by an
+    # installed copy has no other way to name the revision it ran against.
+    "src/openfeature/contrib/tools/provider_tck/spec_revision.json",
 ]
 
 [tool.hatch.build.hooks.custom]

tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py

@@ -76,14 +76,17 @@ def tck_config():
     canonical_flag_set,
     canonical_flags_json,
 )
+from .report import REPORT_DIR_ENV, SCHEMA_VERSION, Outcome
 from .state import TckState
 
 __all__ = [
     "CHANGING_FLAG_KEY",
     "DECLARABLE_CAPABILITIES",
     "DEFAULT_CONFIGURATION",
     "EXTENSIONS_DIRECTORY",
+    "REPORT_DIR_ENV",
     "RESERVED_CAPABILITIES",
+    "SCHEMA_VERSION",
     "BackendControl",
     "Capability",
     "ConnectionControl",
@@ -92,6 +95,7 @@ def tck_config():
     "HttpControl",
     "InProcessControl",
     "KnownDeviation",
+    "Outcome",
     "TckConfig",
     "TckState",
     "UnsupportedControlError",