feat(tck): refuse a capability this SDK cannot express, rather than leaving it to adopters

Re-pin the spec submodule to 89b1519a, which adds a fifth rule for declaring to
Appendix F -- a capability the language's SDK cannot express is refused by the
implementation, not left to adopters -- and corrects the canonical flag set's
comments, which still told every reader that every scenario expects reason
STATIC. Neither feature file changes.

Two such capabilities exist anywhere: @large-integers where the integer accessor
is a 32-bit Integer, and @numeric-coercion where the language has one numeric
type and "a float requested as an integer" does not name two different requests.
Neither says anything about a provider. Leaving it to adopters means every
adopter in the language has to know a fact about their language and remember to
act on it, and in one implementation three separate suites each left the same
capability undeclared with its own comment restating the same property -- three
places to get right, and a single wrong one puts a claim in a report that no
scenario could have verified.

So INEXPRESSIBLE_CAPABILITIES maps such a capability to the property of the SDK
that puts the question out of reach, TckConfig refuses one at construction, the
capability gate skips its scenarios with that reason, and a knownDeviations entry
may not name one -- the gap would be the language's and the entry would attribute
it to this provider. A mapping rather than a set because the message has to name
the property: an adopter who reaches this has done nothing wrong and "the
specification says you may not" is not something they can act on.

The two refusals stay distinguishable, in separate predicates with separate
messages and separate skip reasons. A reservation is global and temporary -- no
scenario anywhere carries the tag, and it expires the moment the specification
writes one. An inexpressibility is one language's and permanent: the scenarios
exist and other languages run and pass them. A reader seeing a capability absent
from a report has to be able to tell "this provider declined" from "no provider
in this language can be asked", because only the first says anything about the
provider. Where a scenario is gated by both kinds, the language-wide reason wins,
because the provider's declaration could not have made that scenario run either
way.

**The mapping is empty in Python, and that was measured rather than assumed.**
`int` is arbitrary-precision; FlagType.INTEGER and FlagType.FLOAT are separate,
reach separate provider methods and are type-checked against `int` and `float`
separately. All four questions the two tags ask were put through the SDK's own
client against a provider implementing the borrowed coercion rule: 2^53 - 1
resolved exactly, 0.5 as an Integer gave TYPE_MISMATCH and the caller's default,
10.0 as an Integer gave 10 and 10 as a Float gave 10.0.

The adoptions agree from the other direction, and this was measured too rather
than reasoned from the source: declaring @numeric-coercion in both flagd suites
and running it, the in-process resolver refuses 0.5 as an integer and widens 10
to a float, while the RPC resolver widens 10 and silently narrows 0.5 to 0. Two
resolvers of one provider giving different answers to the same three questions is
exactly what a language that could not ask them makes impossible. Both are
defects in an implementation, withholding the tag is the honest report for each,
and neither is anything the language prevents. The third scenario fails on both
for a third reason again -- flagd-testbed seeds no integral-float-flag -- which
is also why the run leaves both suites' declarations exactly as they were.

So nothing here is in force, and the machinery is added anyway. The rule belongs
to Appendix F rather than to this package, a future capability may hit it, and
the costs are not symmetric: an unused mechanism is a few lines nobody reads,
while a missing one is discovered by an adopter publishing a claim no scenario
could have examined. It is exercised rather than left dead -- the tests supply
an entry and drive the refusal, the deviation refusal, both skip reasons and the
precedence between them, so a mechanism with no instances is still known to work.

DECLARABLE_CAPABILITIES is derived from both sets rather than listing what it
excludes, which matters precisely because the new one is empty here: a derivation
that quietly dropped it would look right in Python forever and be wrong in the one
language where an entry gets added.

Self-tests move from 214 passed / 42 skipped to 221 passed / 42 skipped.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Author: aepfli

tools/openfeature-tck/README.md

@@ -81,7 +81,12 @@ def tck_config():
 
 import importlib.resources
 
-from .capability import DECLARABLE_CAPABILITIES, RESERVED_CAPABILITIES, Capability
+from .capability import (
+    DECLARABLE_CAPABILITIES,
+    INEXPRESSIBLE_CAPABILITIES,
+    RESERVED_CAPABILITIES,
+    Capability,
+)
 from .compose import (
     DEFAULT_BACKEND_SERVICE,
     DEFAULT_CONTROL_PORT,
@@ -125,6 +130,7 @@ def tck_config():
     "DEFAULT_CONTROL_PORT",
     "DEFAULT_STARTUP_TIMEOUT",
     "EXTENSIONS_DIRECTORY",
+    "INEXPRESSIBLE_CAPABILITIES",
     "RESERVED_CAPABILITIES",
     "BackendControl",
     "BackendEndpoint",

tools/openfeature-tck/spec

@@ -3,7 +3,9 @@
 from __future__ import annotations
 
 import typing
+from collections.abc import Mapping
 from enum import Enum
+from types import MappingProxyType
 
 __all__ = ["Capability"]
 
@@ -24,6 +26,12 @@ class Capability(str, Enum):
     run is worse than no suite at all.
 
     Scenarios with no capability tag are mandatory and always run.
+
+    Two kinds of capability are refused rather than declared, and they are
+    refused for different reasons and with different messages:
+    :data:`RESERVED_CAPABILITIES`, which no scenario anywhere carries yet, and
+    :data:`INEXPRESSIBLE_CAPABILITIES`, whose question this language's SDK cannot
+    put at all. :data:`DECLARABLE_CAPABILITIES` is what is left.
     """
 
     LIFECYCLE = "lifecycle"
@@ -199,8 +207,13 @@ class Capability(str, Enum):
     The SDK's own ``InMemoryProvider`` cannot declare this: it hands values
     back untouched and the client's type check is ``isinstance``-based, so
     ``10.0`` requested as an integer is a ``TYPE_MISMATCH`` rather than ``10``.
-    The width of the integer accessor is a separate property, and a separate
-    capability: :attr:`LARGE_INTEGERS`.
+    That is the provider declining to coerce, not the language refusing to ask:
+    a provider that does coerce returns an ``int`` and the same check passes it.
+    In a language with one numeric type the question could not be put at all,
+    which is why Appendix F names this as inexpressible there and why
+    :data:`INEXPRESSIBLE_CAPABILITIES` is empty here. The width of the integer
+    accessor is a separate property, and a separate capability:
+    :attr:`LARGE_INTEGERS`.
     """
 
     LARGE_INTEGERS = "large-integers"
@@ -214,7 +227,10 @@ class Capability(str, Enum):
 
     Python's ``int`` is unbounded, so a Python provider declares it unless
     something of its own -- a 32-bit field in its wire format, a float on the
-    way through -- narrows the value. Nothing above 2^53 - 1 is asked for:
+    way through -- narrows the value. Which makes this one of the two
+    capabilities Appendix F names as inexpressible somewhere and **not** here:
+    :data:`INEXPRESSIBLE_CAPABILITIES` is empty in Python, and says on what
+    measurement. Nothing above 2^53 - 1 is asked for:
     JavaScript cannot represent it, and what a provider owes a value that does
     not fit the requested accessor is the open question in
     `open-feature/spec#430 <https://github.com/open-feature/spec/issues/430>`_.
@@ -394,6 +410,25 @@ def reserved(self) -> bool:
         """Whether this capability exists in the vocabulary but gates no scenario."""
         return self in RESERVED_CAPABILITIES
 
+    @property
+    def inexpressible(self) -> bool:
+        """Whether this SDK cannot put the question this capability's scenarios ask.
+
+        Distinct from :attr:`reserved` in every respect except that both end in a
+        refusal. See :data:`INEXPRESSIBLE_CAPABILITIES`.
+        """
+        return self in INEXPRESSIBLE_CAPABILITIES
+
+    @property
+    def inexpressible_reason(self) -> str | None:
+        """Which property of this SDK puts the question out of reach, or ``None``.
+
+        The property, not the rule: a message that only says "this cannot be
+        declared" leaves the adopter to discover why, and the why is the part
+        they could not have been expected to know.
+        """
+        return INEXPRESSIBLE_CAPABILITIES.get(self)
+
     def __str__(self) -> str:
         return self.tag
 
@@ -418,10 +453,58 @@ def __str__(self) -> str:
 capability that *can* be verified and is refused the chance.
 """
 
+INEXPRESSIBLE_CAPABILITIES: Mapping[Capability, str] = MappingProxyType({})
+"""Capabilities this language's SDK cannot put the question for, and why.
+
+**Empty in Python, and that is a measurement rather than an omission.** The two
+that exist anywhere are :attr:`Capability.LARGE_INTEGERS`, inexpressible where
+the integer accessor is a 32-bit ``Integer``, and
+:attr:`Capability.NUMERIC_COERCION`, inexpressible where the language has a
+single numeric type and "a float requested as an integer" does not name two
+different requests. Python has neither property: ``int`` is arbitrary-precision,
+and ``get_integer_details`` and ``get_float_details`` are separate accessors
+reaching separate provider methods, type-checked against ``int`` and ``float``
+separately. Both were checked by asking all four questions through the SDK
+rather than by reading its source, and every one of them was answered.
+
+So this mapping carries no entries, and the machinery around it carries no load
+here. It exists anyway because the rule is Appendix F's rather than this
+package's, because the next capability may hit it, and because the cost of the
+two is not symmetric: an unused mechanism is a few lines nobody reads, while a
+missing one is discovered by an adopter publishing a claim no scenario could
+have examined.
+
+**Not the same thing as a reservation, and the difference is what the two
+messages have to carry.** A reserved capability is global and temporary -- no
+scenario anywhere carries the tag, and the reservation expires the moment the
+specification writes one. An inexpressible capability is one language's and
+permanent: the scenarios exist, other languages run them and pass them, and
+nothing changes until the SDK does. A reader seeing a capability missing from a
+report has to be able to tell *"this provider declined"* from *"no provider in
+this language can be asked"*, because only the first says anything about the
+provider. Hence a mapping rather than a set: the value is the property of the
+SDK that puts the question out of reach, and it is the half of the message an
+adopter could not have worked out for themselves.
+
+A capability belongs here only when **no** provider in this language could ever
+satisfy it. A provider that gets the answer wrong is a different thing entirely
+and belongs nowhere near this mapping: flagd's Python provider fails one of the
+three ``@numeric-coercion`` scenarios and passes the other two, which is a
+defect in one implementation, recorded where that adoption records its defects.
+Withholding the tag is the honest report for it. Listing it here would say the
+question cannot be asked, and the other two scenarios passing is the proof that
+it can.
+
+Never overlaps :data:`RESERVED_CAPABILITIES`: a tag no scenario carries is
+reserved, whatever any SDK could express about it.
+"""
+
 DECLARABLE_CAPABILITIES: frozenset[Capability] = (
-    frozenset(Capability) - RESERVED_CAPABILITIES
+    frozenset(Capability)
+    - RESERVED_CAPABILITIES
+    - frozenset(INEXPRESSIBLE_CAPABILITIES)
 )
-"""Every capability an adoption may declare: the vocabulary minus the reserved tags.
+"""Every capability an adoption may declare: the vocabulary minus what is refused.
 
 A reasonable starting point for a new adoption: declare everything, run the
 suite, and remove only what the provider genuinely cannot do. Narrowing from this
@@ -434,6 +517,12 @@ def __str__(self) -> str:
 way past, which is how one implementation came to report ``@targeting`` and
 ``@caching`` as declared without anyone deciding to claim them -- back when both
 were reserved.
+
+It excludes :data:`INEXPRESSIBLE_CAPABILITIES` for the same reason and one more:
+that set is empty in Python, so a default spanning the whole enum would look
+correct here forever and be wrong the day an entry is added, in the one language
+where it was added. Derived rather than listed, so it cannot be the thing that is
+out of date.
 """
 
 _BY_MARKER: dict[str, Capability] = {c.value: c for c in Capability}

tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py

@@ -89,7 +89,11 @@ class KnownDeviation:
     to no capability.
 
     A reserved capability is refused: no scenario carries the tag, so there is
-    nothing to deviate from. See :data:`~.capability.RESERVED_CAPABILITIES`.
+    nothing to deviate from. See :data:`~.capability.RESERVED_CAPABILITIES`. So
+    is one this SDK cannot express, for the opposite reason -- the scenarios
+    exist and no provider here can attempt them, so the gap is the language's
+    and not this provider's. See
+    :data:`~.capability.INEXPRESSIBLE_CAPABILITIES`.
     """
 
     @classmethod
@@ -214,11 +218,14 @@ class TckConfig:
     Naming a reserved capability here is rejected at construction rather than
     passed into a report. See :data:`~.capability.RESERVED_CAPABILITIES`.
 
-    A capability that cannot hold in a language at all -- ``@numeric-coercion``
-    where the language has a single numeric type, ``@large-integers`` on a
-    32-bit accessor -- is a property of the SDK rather than of the provider, and
-    Appendix F records it once rather than every report restating it. Here it is
-    simply left undeclared, and the skip carries the reason.
+    So is one this language's SDK cannot put the question for at all --
+    ``@numeric-coercion`` where the language has a single numeric type,
+    ``@large-integers`` on a 32-bit accessor. That is a property of the SDK
+    rather than of the provider, so it is refused here rather than left for
+    every adopter to know and remember, and the error names the property. The
+    two refusals are deliberately not the same message, and the scenarios they
+    skip do not carry the same reason: see
+    :data:`~.capability.INEXPRESSIBLE_CAPABILITIES`, which is empty in Python.
     """
 
     known_deviations: Sequence[KnownDeviation] = ()
@@ -283,6 +290,7 @@ def __post_init__(self) -> None:
         object.__setattr__(self, "known_deviations", tuple(self.known_deviations))
 
         problems.extend(reserved_problems(self.capabilities))
+        problems.extend(inexpressible_problems(self.capabilities))
         problems.extend(deviation_problems(self.known_deviations))
 
         if (
@@ -353,6 +361,51 @@ def reserved_problems(declared: Iterable[Capability]) -> list[str]:
     ]
 
 
+def inexpressible_problems(declared: Iterable[Capability]) -> list[str]:
+    """Refuse a capability this language's SDK cannot put the question for.
+
+    Refused here rather than left to adopters, because leaving it to adopters
+    means every adopter in the language has to know a fact about their language
+    and remember to act on it. Three suites in one implementation each left the
+    same capability undeclared with its own comment restating the same property
+    of the language: three places to get right, every one of them re-paid by the
+    next adoption, and a single wrong one puts a claim in a report that no
+    scenario could have verified. Appendix F makes this the implementation's job
+    for exactly that reason.
+
+    **The message names the property of the SDK, not the rule.** An adopter who
+    reaches this has done nothing wrong -- they declared a capability their
+    provider may well have -- so the error has to tell them something they could
+    not have known, and "the specification says you may not" is not it.
+
+    Separate from :func:`reserved_problems` on purpose, and it stays separate
+    even though both end in the same refusal. A reserved capability is global and
+    temporary: nothing anywhere carries the tag, and the reservation expires when
+    the specification writes a scenario. An inexpressible one is this language's
+    and permanent: the scenarios exist and other languages pass them. Collapsing
+    them into one predicate would make the two indistinguishable at the only
+    moment anybody is looking.
+    """
+    refused = [
+        capability
+        for capability in declared
+        if isinstance(capability, Capability) and capability.inexpressible
+    ]
+    if not refused:
+        return []
+    return [
+        f"{capability.tag} cannot be declared in this language: {reason}. No "
+        f"provider in this SDK can be asked the question its scenarios put, so a "
+        f"declaration could not be verified either way, and its absence from a "
+        f"report says nothing about your provider. Its scenarios are skipped with "
+        f"that reason. This is not a reservation -- the scenarios exist and other "
+        f"languages run them -- and there is nothing for you to fix; it changes "
+        f"when the SDK does"
+        for capability in sorted(refused, key=lambda c: c.tag)
+        if (reason := capability.inexpressible_reason) is not None
+    ]
+
+
 def deviation_problems(deviations: Sequence[KnownDeviation]) -> list[str]:
     """Refuse a deviation that says nothing a consumer can use.
 
@@ -398,6 +451,15 @@ def deviation_problems(deviations: Sequence[KnownDeviation]) -> list[str]:
                 f"could show the gap. Remove it, or name the capability whose "
                 f"scenarios the gap actually affects"
             )
+        elif capability.inexpressible:
+            problems.append(
+                f"known_deviations[{index}] names {capability.tag}, which cannot "
+                f"be expressed in this language: {capability.inexpressible_reason}. "
+                f"A deviation says this provider fails something it is required to "
+                f"do, and no provider in this SDK can attempt these scenarios at "
+                f"all -- so the entry would attribute to your provider a gap that "
+                f"belongs to the language. The skip already carries that reason"
+            )
 
     return problems
 

tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py

@@ -20,6 +20,7 @@
 from __future__ import annotations
 
 import typing
+from collections.abc import Iterable
 from pathlib import Path
 
 import pytest
@@ -212,6 +213,16 @@ def tck_state(tck_config: TckConfig) -> typing.Iterator[TckState]:
 
 @pytest.fixture(autouse=True)
 def _tck_capability_gate(request: pytest.FixtureRequest) -> None:
+    """Autouse wrapper around :func:`capability_gate`.
+
+    A one-line fixture over a plain function, so the decision it makes can be
+    put under test without reaching inside a fixture object for the callable
+    pytest wrapped -- which is private, and has moved between pytest versions.
+    """
+    capability_gate(request)
+
+
+def capability_gate(request: pytest.FixtureRequest) -> None:
     """Skip a scenario whose capability the provider did not declare.
 
     ``pytest.skip`` here reports the scenario as skipped **with the reason**,
@@ -226,6 +237,17 @@ def _tck_capability_gate(request: pytest.FixtureRequest) -> None:
 
     Checking markers first also means the gate costs nothing, and instantiates
     nothing, for tests that are not TCK scenarios.
+
+    **A capability this SDK cannot express is skipped first, and says so.** Its
+    scenarios would be skipped anyway -- nothing may declare it, so nothing
+    does -- but with the wrong reason. "The provider does not declare it" reads
+    as a decision the provider made, and no provider in this language had one to
+    make; a reader of the report has to be able to tell those apart, because only
+    the first says anything about the provider. Checked before the declaration
+    loop rather than inside it so that a scenario gated by both kinds reports the
+    permanent, language-wide reason rather than whichever tag came first off the
+    marker iterator. Empty in Python; see
+    :data:`~.capability.INEXPRESSIBLE_CAPABILITIES`.
     """
     gated = [
         capability
@@ -235,17 +257,59 @@ def _tck_capability_gate(request: pytest.FixtureRequest) -> None:
     if not gated:
         return
 
+    inexpressible = inexpressible_skip_reason(gated)
+    if inexpressible is not None:
+        pytest.skip(inexpressible)
+
     try:
         config: TckConfig = request.getfixturevalue("tck_config")
     except pytest.FixtureLookupError:
         return
 
+    undeclared = undeclared_skip_reason(gated, config)
+    if undeclared is not None:
+        pytest.skip(undeclared)
+
+
+def inexpressible_skip_reason(gated: Iterable[Capability]) -> str | None:
+    """Why these scenarios cannot be run in this language at all, or ``None``.
+
+    Says nothing about the provider, and says so, because the alternative
+    reading is the one a reader will reach for: a capability missing from a
+    report usually means the provider declined. Here nothing declined -- no
+    provider in this SDK could be asked -- and Appendix F makes telling those
+    two apart the implementation's job rather than the reader's.
+
+    Deterministic when more than one applies: the tags are sorted, so the
+    message does not depend on the order markers come off a node.
+    """
+    for capability in sorted(gated, key=lambda c: c.tag):
+        reason = capability.inexpressible_reason
+        if reason is not None:
+            return (
+                f"{capability.tag} cannot be expressed by this SDK, so no provider "
+                f"in this language can be asked: {reason}. Nothing about the "
+                f"provider under test follows from this skip"
+            )
+    return None
+
+
+def undeclared_skip_reason(
+    gated: Iterable[Capability], config: TckConfig
+) -> str | None:
+    """Why this provider is not being asked these scenarios, or ``None``.
+
+    The other half of the pair, and the one that *is* about the provider: it
+    declined, and the declaration it did make is quoted so a reader can see what
+    was claimed instead.
+    """
     for capability in gated:
         if not config.declares(capability):
-            pytest.skip(
+            return (
                 f"provider does not declare capability {capability.tag}. "
                 f"Declared: {' '.join(config.sorted_capabilities) or '(none)'}"
             )
+    return None
 
 
 @pytest.fixture(scope="session", autouse=True)