feat: add Appendix F, provider conformance (TCK) - #423
Draft
aepfli wants to merge 52 commits into
Draft
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
aepfli
added a commit
to open-feature/java-sdk-contrib
that referenced
this pull request
Aug 24, 2026
…pec submodule
The Gherkin, the canonical flag set and the control API document are not Java
artifacts. They are language-agnostic definitions of the provider contract that
every language's TCK must agree on byte for byte, and they only lived in this
module because the proof of concept had to start somewhere.
They now live in open-feature/spec as Appendix F, under
specification/assets/provider-tck/, and are copied in from the `spec` git
submodule at generate-resources — the same mechanism tools/flagd-api-testkit
already uses for the flagd test harness. The copies are git-ignored and carry a
do-not-edit note; changes belong in the spec repo and arrive here by bumping the
submodule.
Consumers are unaffected: the artifacts are still packaged into the release JAR,
@SelectClasspathResource("features") still resolves, and nobody needs a submodule
of their own. Verified byte-identical after the round trip.
The in-memory CI job now checks out submodules, since without them there is no
suite to run.
DEPENDS ON open-feature/spec#423. The submodule is pinned to that PR's branch
commit rather than to a commit on the spec repo's main branch. That is reachable,
so CI can fetch it, but it must be re-pinned to main once #423 merges and before
this lands.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli
added a commit
to open-feature/java-sdk-contrib
that referenced
this pull request
Aug 24, 2026
…pec submodule
The Gherkin, the canonical flag set and the control API document are not Java
artifacts. They are language-agnostic definitions of the provider contract that
every language's TCK must agree on byte for byte, and they only lived in this
module because the proof of concept had to start somewhere.
They now live in open-feature/spec as Appendix F, under
specification/assets/provider-tck/, and are copied in from the `spec` git
submodule at generate-resources — the same mechanism tools/flagd-api-testkit
already uses for the flagd test harness. The copies are git-ignored and carry a
do-not-edit note; changes belong in the spec repo and arrive here by bumping the
submodule.
Consumers are unaffected: the artifacts are still packaged into the release JAR,
@SelectClasspathResource("features") still resolves, and nobody needs a submodule
of their own. Verified byte-identical after the round trip.
The in-memory CI job now checks out submodules, since without them there is no
suite to run.
DEPENDS ON open-feature/spec#423. The submodule is pinned to that PR's branch
commit rather than to a commit on the spec repo's main branch. That is reachable,
so CI can fetch it, but it must be re-pinned to main once #423 merges and before
this lands.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
This was referenced Aug 24, 2026
aepfli
added a commit
to open-feature/java-sdk-contrib
that referenced
this pull request
Aug 24, 2026
lifecycle.feature was tagged @events, which is wrong in both directions.
Too strict: a stateless provider such as OFREP emits no events of its own
and cannot declare @events, yet it still initialises against a backend and
still owes the lifecycle contract.
Too lax: FeatureProviderStateManager emits PROVIDER_READY/PROVIDER_ERROR
around initialize for any provider, whether or not it is an EventProvider.
So a provider that does no initialisation of its own reaches READY exactly
as NoOpProvider would, and the readiness scenario passes vacuously.
Adds Capability.LIFECYCLE ("@lifecycle") -- performs an initialisation that
reaches its backend, with an observable outcome -- and re-vendors
lifecycle.feature verbatim from the spec assets, where the feature-level tag
is now @lifecycle (spec dfa16586, PR open-feature/spec#423).
flagd declares LIFECYCLE in both resolver modes: RPC does a round trip and
in-process syncs the whole ruleset during initialisation, so the scenarios
assert something real there.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli
added a commit
to open-feature/js-sdk-contrib
that referenced
this pull request
Aug 24, 2026
…odule The feature files, canonical flag set and control-API document are owned by open-feature/spec. Vendoring them here made this repository a second place the definition of conformance could drift, which is precisely what the suite exists to prevent. They are now a pinned submodule at libs/shared/provider-tck/spec and the copies are gone. Adopters are unaffected, and that is the constraint the change had to respect: the rollup asset globs copy the artifacts out of the submodule and into the published package, so installing @openfeature/provider-tck from npm still needs no submodule and no particular repository layout. resolveAssetDir therefore has to satisfy two layouts -- the packaged copy next to the bundle, and the submodule under the library root -- and tries both. The spec calls the feature directory `gherkin`; the package keeps the name the API talks about. Contributors do need the submodule: without it no feature file loads at all. `nx test` and `nx package` depend on a pullSpec target that initialises it, and CI already checks out with `submodules: recursive`. Prettier is pointed at the submodule instead of the old vendored paths so it never rewrites artifacts that are consumed byte for byte by every language's TCK. The .gitattributes normalising those files to LF goes with them; the equivalent lives upstream, where the files now do. Pinned to dfa1658 (open-feature/spec#423). Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli
force-pushed
the
feat/provider-tck-appendix
branch
from
August 24, 2026 14:08
4661068 to
36392a9
Compare
This was referenced Aug 24, 2026
toddbaert
reviewed
Sep 9, 2026
| @@ -0,0 +1,82 @@ | |||
| { | |||
Member
There was a problem hiding this comment.
I wonder if the canonical flags should also be gherkin? They could be a very large "given", I guess.
Member
Author
There was a problem hiding this comment.
i can see the motivition behind it. but i am not sure it will be easier to read or understand or parse. Especially a json is easily transformed into another JSON for eg. CLI's to prefill databases etc. Not so sure about the gherkin for this purpose.
toddbaert
reviewed
Sep 9, 2026
Comment on lines
+30
to
+45
| Scenario: An integer flag resolves as an integer | ||
| # Paired with the float scenario below and with the narrowing scenario in errors.feature. | ||
| # Together they pin down that the two numeric types stay distinct rather than both being | ||
| # funnelled through one numeric representation. | ||
| Given a Integer-flag with key "integer-flag" and a default value "1" | ||
| When the flag was evaluated with details | ||
| Then the resolved details value should be "10" | ||
| And the error-code should be "" | ||
| And no exception should have been thrown | ||
|
|
||
| Scenario: A float flag resolves as a float | ||
| Given a Float-flag with key "float-flag" and a default value "0.1" | ||
| When the flag was evaluated with details | ||
| Then the resolved details value should be "0.5" | ||
| And the error-code should be "" | ||
| And no exception should have been thrown |
Member
There was a problem hiding this comment.
What about integer precision? I think a lot of our SDKs support resolutions above 32 bit (though some like JS are practically capped at 2⁵³... it might be worth testing at least up to 2⁵³ -1
Member
Author
There was a problem hiding this comment.
fully agree, but for now, i would love to focus on the method, rather than the tests. We should agree and finalize the basic process. Afterwards we can add more tests to ensure compatibility
Member
Author
There was a problem hiding this comment.
added more cases on the side
toddbaert
reviewed
Sep 9, 2026
toddbaert
self-requested a review
September 9, 2026 22:48
Member
|
I have to say, the Java impl is remarkably small and clean in Java. I think there's a lot of value here. |
Member
|
I really like the idea, and the impl is clean (though TBH the issue and PR description is very wordy and maybe could be compressed and made less verbose) it took me a long time to get through 😅 . I have 2 things I think I'd want to understand additionally:
|
Member
Author
|
I am in total favor for extensibility. This would even help our providers. We could try to define defaults in java, like a default resource path and a default step path, add those to the test, and have a simply way for the beginning. But also I am not sure, if this would be something which we should delay to the next version. For now, there is the idea, but do we need it immediately to ensure conformance. It sounds like a nice to have feature. A good iteration on the tck, i feel like it could bloat the current efforts. wdyt? I measured what the Java classpath actually does here — the defaults idea works, with one catchFour measurements against the real module:
The third is the catch. If the default resource path is just plus one extra glue package on the base suite ( Net cost: two lines in If it does land, I'd word it in Appendix F as an outcome rather than a mechanism — an adopter can add scenarios and steps that run in the same backend phase without redeclaring the canonical set — since it's pure convention in Java and Python, while Go and JS have no runtime scanning and need one registration callback. Worth separating from the above either way: the suite should verify its canonical scenarios actually ran. That is not about extensibility — the same footgun already exists as // edit: flagd specifically is actually not a good reference example, as we are testing the evaluation engine seperately now. with really good tests. not sure if we want to have all of those tests executed during TCK evaluation. I see the benefit and the ease of running. But the more i think about, the TCK might be best to kept seperate as a verifiable unit, with whom nobody can mingle. Still torn, by both sides |
This was referenced Sep 10, 2026
Closed
aepfli
force-pushed
the
feat/provider-tck-appendix
branch
from
September 11, 2026 07:27
4c88e46 to
2acd860
Compare
…rowing I invented targeted-flag. flagd-testbed already serves targeting-key-flag with the same rule shape -- variants hit and miss, defaultVariant miss, a rule on the targeting key -- exercised by its own targeting.feature and present in both its flag sets. Adopting it instead of a new flag is the same reasoning that named the falsy flags after Appendix B's: a backend serving that harness already serves this one, so the canonical set costs it nothing. It also removes an upstream dependency this change would otherwise have had, since nothing needs adding to the testbed at all. Its own scenario settles a question these scenarios had to guess at. flagd reports TARGETING_MATCH for the hit and DEFAULT for the miss, so with a rule present but unmatched there is no single reason to assert. Had these scenarios pinned STATIC, as every other evaluation scenario does, the non-matching one would have failed against flagd. They assert values, which carry the whole signal. That leaves the reason assertions elsewhere as a narrowing of the specification, and this states it rather than leaving it to be discovered. 2.2.5 is a SHOULD that explicitly permits "some other string indicating the semantic reason", and the suite pins an exact value in thirty-one scenario instances. It stays that way for now because the reason is the cheapest diagnosis of silent fallback, and because gating it would mean a second capability that nearly every provider would declare anyway. What is written down is that a provider failed by it is not thereby defective, and that the fix is to revisit this decision rather than to record a deviation against that provider. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The extension rule recounted a real incident -- a selector matching one scenario name produced a green suite and a well-formed report -- and gave the set size as twenty-nine. That was true when it was written. Three count changes later it reads as a claim about the current set, and Java's TCK now counts fifty-two scenario instances against twenty-seven definitions, so any number here is a third place for the same fact to drift. The anecdote's force never depended on the figure: one scenario out of the whole set is the point. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…tecture A flag disabled in the management system resolves to nothing and the caller's default stands in. No scenario covered it. Appendix B does, for SDKs, and gates it behind @reason-codes-disabled -- and the only trace in the provider section is DISABLED appearing in 2.2.5's list of reason strings. Requirement 1.4.7 is about the SDK propagating whatever reason arrived, not about what a provider owes a disabled flag. So this appendix states the behaviour, as it does for @numeric-coercion, and gates it. Gated rather than mandatory because the substitution happens in different places. A provider that evaluates locally -- flagd's two resolvers, an in-memory provider -- can return the value the caller passed in. A provider whose backend decides, such as one speaking OFREP, cannot: the server never sees the caller's default. The same flag cannot behave the same way across those two, and neither is wrong. That is what a capability is for. The scenarios assert the value and the absence of an error, not the reason. Each row's caller default differs from the flag's configured value, so a provider that ignores the state returns the configured value and fails on the value alone, which rests on 2.2.3, a MUST. Pinning reason DISABLED would rest on 2.2.5, a SHOULD that permits "some other string", and would be the seventh place this suite asserts more than the specification states. No variant is asserted: a disabled flag has resolved no variant, so @disabled-flags and @Variants deliberately do not compose. There is no disabled-object-flag for the same reason in reverse -- an Object row would need @object as well, and one row of an outline cannot carry its own tag. Four flags, mirroring boolean-flag, string-flag, integer-flag and float-flag and differing only in state. Names, variants and values are flagd-testbed's own from flags/disabled-flags.json, which the launchpad already serves, so nothing needs adding upstream -- the same reasoning that made targeting-key-flag free. Two statements this falsifies are corrected: the canonical set no longer has every flag resolving to its default variant, and ENABLED is no longer universal. Both are now stated as the load-bearing properties they are, in the appendix and in the asset README. Scenario instances go 52 to 56. Note for the language suites: a test that iterates the packaged flags and asserts each resolves to its own defaultVariant now has four exceptions. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The comment said a provider whose backend decides "cannot" substitute the caller's default, "such as one speaking OFREP", because the server never sees that default. The conclusion was right and the reasoning was wrong, and three implementations disproved it within a day of it being written. flagd's RPC resolver is remote by exactly that description and passes: the Python provider substitutes locally when the reason is DISABLED and no variant came back (resolvers/grpc.py). The Go and JavaScript OFREP providers pass too, on the same signal -- OFREP's codeDefaultFlag schema omits `value` precisely so the provider can use the code default, and says so normatively. Where evaluation happens is not the axis. What the axis actually is: the backend has to distinguish a disabled flag at all, and the provider has to substitute on the strength of that signal. Either half can be missing. Both OFREP providers that fail today fail on the second half and for unrelated reasons -- one returns FLAG_NOT_FOUND for any code-default response, the other indexes an optional field unconditionally -- which is what a capability is supposed to separate from a defect. Only the comment changes. No scenario, flag or assertion is touched, and the scenario count stays at 56. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
… stack Two things four implementations answered differently, which is one more answer than a portable conformance claim can carry. A knownDeviations entry is legitimate in two shapes -- declare the capability and let the scenario fail, or withhold it and explain the absence -- and the appendix said nothing about which. Java and Go documented the second, JS the first and called the second discouraged, Python required an issue link and had no untracked form at all. A consumer comparing four reports would read one field three ways. The first shape is now preferred, in prose, with the reason: withdrawing a capability to turn a failure into a skip hides a defect behind something that looks deliberate, which is the outcome the field exists to prevent rather than one of its uses. summary is required, issue is not. "Own the lifecycle" already covered the container stack, and three of four implementations shipped only the control-API client anyway -- so every flagd adoption hand-rolled an orchestrator, between 130 and 460 lines of it. That satisfies the letter of the item and not its point, so the item now says what an adopter supplies (a Compose file, which service and ports, a factory taking an endpoint) and what the suite owes. The start-once rule and the ask-do-not-sleep rule are stated with their reasons, because both were learned by getting them wrong. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…not guess Appendix F described the in-process allowance and the rule that a provider with an external backend must use the control API, but said nothing about how a reader finds out which one a given run actually used. Four implementations filled that silence four different ways, one of them by inferring it from the control's concrete type -- which is correct for the two controls the suite ships and silently wrong for an adopter's own, the only case where the question is open. So: the control states it, it is required, and an omitted value is not neutral. Every run is one or the other, which makes silence an unfalsifiable claim rather than an abstention. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
… returns Only /start carried "MUST NOT return until the seeded flag state is actually being served". /change and /reset change flag state too and said nothing, so the promise a caller can rely on depended on which endpoint it called -- and /reset is the one a TCK calls before every scenario, which is the worst place for the gap to be. A window there is re-rolled per scenario instead of once per suite, so a small per-call probability becomes a near-certainty over a few hundred scenarios and the failures land on scenarios chosen at random. Two runs then disagree about which scenarios failed, which reads as a flaky provider and is the hardest shape of this to diagnose. For /change the promise is spelled out against the backend rather than the provider: a fresh evaluation resolves the new value once the call returns, while how long the provider takes to notice is its transport's business and is what the event timeout measures. Conflating the two makes detection latency unmeasurable, because the clock would start before there is anything to detect. The appendix gains the invariant in prose beside the two it already lists, plus where a wait belongs when an adopter is stuck with a backend that breaks it: in that adoption, named and citing the defect, never as a constant in the shared harness where every later adopter inherits it without knowing why. The implementer checklist's readiness note is corrected to match -- it said to ask the control API rather than sleep, which is right for the stack coming up and wrong as a general instruction, since after that the endpoints owe the caller the wait themselves. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The endpoint was marked [REQUIRED] on the strength of a claim in its own description -- "the TCK uses this for the disconnect/reconnect scenarios" -- that was not true. The disconnect/reconnect scenario is written as an unbounded outage, "the connection is lost" then "the connection is restored", which an implementation serves with /stop followed by /start. Nothing in the shipped Gherkin reaches /restart in any of the four languages: Go dropped its binding deliberately, and the Java and Python clients kept one that no step can call. So the contract was asking every backend author to implement an endpoint nothing invoked. Demoted rather than deleted, with the condition that would bring it back written down. /start on reconnect resets flag state where /restart would have preserved it, and that is only acceptable because the one scenario involved asserts events and client state and never a resolved value. A scenario that asserts what a stale provider serves during an outage -- last-known-value caching, held behind @caching and not yet written -- needs exactly this endpoint's preservation. A backend author who wants that capability testable later should implement it now. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…reasons Two cross-language lessons that were living in four READMEs, which is where the known-deviation and control-path decisions had already drifted into three different answers. An adoption suite is not a required gate while real gaps remain: its honest output is red, because it fails on filed provider defects, missing backend fixtures and undeclared capabilities. Making it block a merge forces someone to silence it, and the cheapest way to silence a conformance suite is to stop asking the question -- withdraw a capability, delete an assertion, pin an older backend. So exclude it and say so, with two observed mistakes named: an exclusion that a profile, target or build tag undoes, and an exclusion nobody wrote down. All four implementations believed their suites were excluded; all four were running them, red and unwatched, each defeated by a different mechanism. Keep the suite compiling in the default build even when it does not run, because a suite that has silently stopped building against its harness is worse than one that runs and fails. The caching gap gains the constraint a scenario author needs. A provider may cache client-side and rewrite the reason when it does -- flagd's RPC resolver runs an LRU cache by default and reports CACHED on a repeat evaluation -- so any scenario evaluating the same flag twice sees a different reason the second time from a provider that is behaving correctly. Exactly one current scenario evaluates twice, and it changes the configuration in between. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
This was referenced Sep 12, 2026
…ouse rule The suite asserted an exact reason in thirteen places across three feature files, which narrowed 2.2.5 into a MUST for every adopter. 2.2.5 is a SHOULD that goes further than the others: a provider may use one of the listed values "or some other string indicating the semantic reason for the returned flag value". A provider whose backend reports vendor-specific reasons was therefore being failed for something the specification permits, and the appendix recorded that as a deliberate exception. It bought very little. Every canonical flag resolves to a value distinct from the caller's default, so a provider that silently falls back is already caught by the value assertion -- the reason only said why it failed rather than that it did. And of the thirteen assertions, five sat beside an error-code assertion that already carries the MUST, while the remaining eight asserted STATIC, which is the one reason the specification genuinely leaves open: types.md types DEFAULT as "no dynamic evaluation occurred or dynamic evaluation yielded no result", and a rule-less flag satisfies that as readily as STATIC does. So the reasons move to reason.feature, gated as a whole on @standard-reasons. The gate is a claim rather than an exemption: a provider declares that it uses the standard vocabulary with the standard meanings, and that file checks it. One that does not declare it loses nothing, since values, variants and error codes are asserted everywhere else on MUSTs. What the declaration buys is something a report's reader can act on -- anyone building telemetry or debugging on reason can see the vocabulary was verified rather than assumed. This also lets the appendix state what each reason means without asking the specification to close the set, because the mapping is the content of an opt-in claim and constrains nobody who does not make it. TARGETING_MATCH and DISABLED compose with @targeting and @disabled-flags, since neither reason can be observed without the behaviour that produces it. CACHED and STALE are left out: both need a scenario shape the suite does not have yet. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The four reasons a provider produces in normal execution rest on Requirement 1.4.7, which makes the SDK propagate the provider's reason -- but 1.4.7 is scoped to "cases of normal execution". Abnormal execution is 1.4.9, a SHOULD on the SDK to "indicate an error", and nothing requires the provider's reason to survive it. So an ERROR scenario in a provider suite can pass on a value the SDK wrote, which is the vacuous shape this appendix warns about everywhere else. Kept rather than dropped, because the assertion is the pair and not the field. The error code on its own is already covered for every provider by 2.2.7 and 1.4.8, both MUSTs on a closed enumeration, asserted ungated in errors.feature. The reason on its own could be the SDK's. Their agreement is what neither can satisfy alone, and an evaluation reporting FLAG_NOT_FOUND with reason STATIC is incoherent whoever wrote it. Recorded in both places a reader might look: beside the scenarios, and in the appendix's reason mapping. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…ation Two capabilities say nothing about any provider: @large-integers where the integer accessor is 32-bit, and @numeric-coercion where the language has a single numeric type. No provider in those languages can satisfy them and none ever will until the SDK changes. Until now that was documentation, and adopters were expected to act on it. They did, repeatedly: in one implementation three separate suites each left the same capability undeclared, each with its own comment restating the same property of the language. A fact about a language was being remembered in three places per language and in every future adoption, and a single wrong one would put a claim in a report that no scenario could have verified -- the failure the reserved-capability rules exist to prevent, reached by another route. So the implementation refuses it at configuration time, using the machinery that already refuses a reserved capability. The two refusals are deliberately kept distinguishable in the skip reason, because they are different facts: a reserved capability is global and expires when the specification adds scenarios, while an inexpressible one is a single language's and permanent. A reader seeing a capability absent from a report has to be able to tell "this provider declined" from "no provider in this language can be asked", since only the first describes the provider. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…s STATIC Two $comment blocks survived the reason change: "Every scenario expects reason STATIC" and "which is what lets every other scenario expect reason STATIC". The appendix was corrected in c342461 and this asset was not, which matters more than the usual stale comment -- the flag set is copied verbatim into four languages' packaged artifacts, so it is what a backend author reads when seeding a harness, and it now described a suite that no longer exists. Both now say what is actually true: only targeting-key-flag has a rule, and the absence of rules elsewhere is what lets a provider declaring @standard-reasons expect STATIC there rather than TARGETING_MATCH. The load-bearing property is unchanged -- it was never really about the reason, it was about not seeding targeting onto anything else. Found by the Java implementation while re-pinning, which is the argument for four implementations in one sentence: the asset is consumed by all of them and was wrong in a way no single language's tests could notice. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…rbids "A provider withholding this capability should say which it is -- a deliberate choice, or a tracked defect -- and a conformance report has knownDeviations for the second." That is withhold-plus-deviate, and a few sections above, the known-deviation rule says to prefer declaring the capability and letting the scenario fail, because a withheld capability plus a deviation asserts a defect at something nobody asked. Two parts of one document disagreeing is bad enough; this was also the part an implementer would read to learn the shape, and two of the four adoptions followed it there. Measurement settles which is right: flagd widens 10 to 10.0 correctly and narrows 0.5 to 0 with no error code, in both resolvers and in more than one language. It does coerce and gets one direction wrong, which is exactly what a skip cannot express. Withholding is still right for a provider that cannot attempt the behaviour at all -- a single numeric type, where the distinction does not exist to get wrong. Also adds the carve-out a TCK implementation's own self-tests need, which the rule did not cover. Those suites run the scenarios against an SDK's in-memory provider as a fixture, produce no report about a third party, and run in the ordinary build where a permanently failing scenario is a broken build rather than a finding -- nobody downstream can act on it, because the fix is an SDK release away. They may withhold, on the condition that the defect is pinned by a test of its own so the skip is not the only record. An adoption has no such licence. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Three implementations answered one question three ways, and the question was never written down: what do you declare when the backend under test cannot serve a flag some scenario needs? Go and JavaScript withheld @large-integers; Python declared it and reported a failure that was the backend's. Python also withheld @numeric-coercion on the same reasoning, and Go withheld it while recording a deviation against it. The rule that resolves all of them: declare when at least one scenario gating the tag can actually be put to the provider, withhold only when none can. @large-integers has one scenario and the reference backend serves no flag for it, so nothing can be established. @numeric-coercion has three and a backend missing one flag can still be asked the other two. That difference is not academic. Declaring @numeric-coercion is how one provider's two resolvers were found to disagree with each other -- one coercing correctly, one silently narrowing 0.5 to 0 -- which six passes of reading the source had not revealed. Withholding it to avoid one fixture failure would have cost both answers. Two consequences are stated with it. A scenario failing because its fixture is absent is not a provider defect and must not be recorded as one, or the report accuses the provider of the backend's gap. And a capability withheld for a backend gap is temporary in a way one withheld by choice is not -- it needs a note saying why, or it outlives its reason. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…s owed As written the rule said "declare when at least one scenario gating it can be put to the provider", with no condition on it. Read literally that forces a declaration wherever the scenarios are reachable -- including where the specification permits a provider to decline outright. @numeric-coercion is defined by no requirement, so a provider that simply does not coerce is entitled to withhold it, and the rule as written would have turned that permitted choice into a manufactured failure. That is the mirror image of the mistake the rule exists to prevent. Found by the Go implementation, whose self-tests withhold @numeric-coercion for exactly that reason while its flagd adoption declares it: memprovider does not coerce by design, flagd does coerce and gets one direction wrong. Two bullets four lines apart, resting on different grounds, and only one of them is rule six's business. So the rule now opens with the condition it always had implicitly: it applies once a provider is attempting the capability. Whether the provider owes an answer at all is Decision 4's question and comes first. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…vice provisional Two changes to the CI section, deliberately separated because one is settled and the other is not. Settled: a conformance suite gets a step of its own rather than being folded into a provider's existing end-to-end suite. The reason is what a failure says, not how long the run takes. A dedicated step going red reports that conformance failed; the same scenarios inside an e2e suite report that a test failed, and a reader has to go and find out which kind. The two also mean different things by failure -- an e2e suite is expected green, so a failure is a regression, while a conformance suite fails scenarios by design wherever a knownDeviation is declared, and that failure is correct output until the defect ships a fix. Sharing one signal between "you broke something" and "this is the known state" ends with somebody silencing the informative half. Four implementations answered this differently and none of them decided it; each landed where its build system pushed it. Not settled: whether the suite can be a required gate. That advice rests on the premise that a conformance run is unavoidably red, which is only true because we have been judging it by "zero failures". Judged instead by whether its results match its declaration, a healthy adoption is green in its steady state with its deviations intact. That is under discussion in #417 and the paragraph now says so, so nobody builds on it before it is decided. The separation advice holds either way. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…s not The paragraph justified a dedicated step with "a dedicated step that goes red reports that conformance failed" -- a property that only exists if something invokes the step. The same section says, provisionally, that nothing should: the suite is excluded from the default build pending the gating discussion. So the argument depended on a condition the surrounding advice forbids, and an audit of the two implementations that already have a separate target found exactly that hole -- both satisfy "do not fold it into the e2e suite", neither has a conformance job, so nothing goes red because nothing runs. Rewritten to argue from what a result means rather than from a failure being observed, and to say plainly that this is about a build target rather than a CI job. The two are easy to conflate. A step no pipeline invokes still earns its place: a maintainer running it by hand gets an unambiguous answer instead of a mixed one, and if the gating question resolves towards running it, the thing to gate on already exists and is already scoped to the right scenarios. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The appendix asked for a step of its own and stopped there, so four implementations put the files in four places and three of them selected the suite by test name: a skip filter on "Conformance", a Surefire pattern on *TckTest.java, a nested directory under the end-to-end suite. Only one filed it as a sibling. Stated as a property rather than a layout, because where files live is idiomatic per language and per repository, while the thing that goes wrong is not. A naming convention works until a test is renamed, and then it fails in the direction that hides the problem: the suite stops being selected, the step goes green having run nothing, and a pull request keeps quoting a tally from the last run that happened. That is the same shape as the CI exclusions every implementation believed it had and did not. The evidence is what each approach needed to be trustworthy. Selecting by name required an AST parser asserting that the set of tests calling the runner equals the set matching the pattern, mutation-tested in both directions -- and a third mutation, because such a guard otherwise passes vacuously the day it goes blind. Selecting by directory required nothing. Nesting inside the end-to-end directory gets its own sentence because it is the same mistake one level down: it says the conformance suite is a kind of end-to-end test, which is precisely what a separate step exists to deny, and where the end-to-end suite is its own module it also drags container libraries into tests that never use them. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Three of the four implementations could, by three different routes, and one of them did: a full adoption suite ran against the previous pin's feature files and reported a tally byte-identical to the run before it. Nothing failed, nothing warned, and it was caught only because someone compared two numbers that should have differed. The cause is the same wherever a copy is involved. Moving a pin updates the recorded revision, not the working tree the build copies from, so the two disagree silently and the copy wins. A rebase does the same thing: it moves the gitlink and leaves the checkout behind. Recorded here because it was recorded nowhere durable -- it surfaced in two languages' working notes during a rewrite and would have been lost with them, which is a poor fate for the only evidence that the failure mode is real. It is also the same shape as the CI exclusions every implementation believed it had and did not: a structural property everyone assumed held, in four places, none of them checked. Says to wire the fetch into the build rather than trusting whoever moves the pin to run a second command, notes that an immutable checksummed dependency avoids the problem outright, and warns that a guard catching one symptom is not a substitute -- a pin that changes only a scenario's content passes every such guard and still tests the wrong thing. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…ation Every other capability is covered somewhere without containers, by an in-memory or controllable provider standing in for the backend, so a regression in the step definitions or the capability gate shows up in an ordinary build. @Stale is not, in any of the four implementations -- faking a provider that loses its backend and regains it needs a control that can disconnect, and the in-process control paths refuse connection operations by design. The consequence is the part worth recording: the stale/ready transition is exercised only by a containerised adoption, which is also the suite most likely to be excluded from a default build. So the one capability with no cheap coverage is the one whose only coverage is the easiest to switch off. Surfaced by the JavaScript implementation while rewriting its README, which had the observation in a code comment and nowhere else -- exactly the case this appendix asked implementers to report rather than let vanish. Naming a way to close it, since a gap with no route out reads as a permanent limitation: a controllable provider that can fake a disconnect, written carefully enough not to become a mock that passes whatever the provider does. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The requirement said the suite must keep compiling in the default build. In at least one language that is impossible and always will be: where the ordinary build compiles what gets published, it is configured for library code -- no test globals, a different module target -- and an adoption that calls the test framework's own functions can never join it. Stated that way the requirement was unmeetable, which makes it a permanent known gap rather than a rule. What was actually wanted is that the adoption is typechecked against the harness by something that runs ordinarily, so a signature change in the harness cannot rot an adoption unnoticed. A typecheck scoped to the adoption satisfies that, and is a further argument for the adoption having a directory of its own -- a directory nothing else occupies is something a typecheck can be pointed at. Both ways of failing it are now named, because they are opposites and both were observed within an hour of each other. Excluding the adoption by path can remove it from the build as well as from the run, silently, since nothing fails when nothing is compiled. Removing an exclusion can pull the adoption into a build that cannot compile it, which at least fails loudly. The cheap check is the same either way and is worth stating: introduce a deliberate compile error in the adoption and confirm the ordinary build rejects it. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The appendix said the step vocabulary was inherited from the flagd test harness "wherever it was already provider-neutral", which implies exceptions and named none. There are two, and they are the entire diff: a stable provider and an unavailable one both carried "flagd" in the step text. Recording them here is what lets four language READMEs stop recording them separately -- it was the one piece of provenance the Java implementation could not find an upstream home for -- and it turns an unfalsifiable claim into a checkable one. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…idation The appendix already said CACHED is unasserted because no scenario repeats an evaluation without a configuration change in between. That clause is load-bearing in a second way it did not state: against a provider with a client-side cache -- flagd's RPC resolver runs an LRU by default -- the evaluation after the change answers from the cache unless invalidation works, so the configuration-change scenarios depend on it and nothing tests it directly. An adoption that does not disable the cache is therefore resting on an untested dependency, and a failure there reads as a configuration-change defect rather than an invalidation one. Not requiring adoptions to disable the cache, because a provider measured as it ships is the more useful measurement -- but the dependency should be visible to whoever reads the failure. Found by the Go flagd adoption, which does not disable the cache. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
A property was added without the heading following it. Noticed independently by the Go and Java implementations, both of which read this file while deciding what their READMEs could stop restating -- which is the use the count matters for: an implementer checking they have covered all of them counts the heading, not the bullets. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Appendix F: Provider Conformance (TCK) — a language-agnostic conformance suite for validating OpenFeature providers, plus the three artifacts it rests on.
Marked experimental and explicitly non-normative. This is a proof of concept and the scenario set is a representative subset, not exhaustive coverage.
Why
OpenFeature's central promise is that swapping providers does not change application behaviour, and nothing currently verifies it. Every provider tests itself differently, against its own backend, with its own harness — so "implements the provider contract" is an unverified claim, and a behavioural difference between two providers gets discovered by the application that trips over it.
Why a new appendix rather than extending Appendix B
They answer different questions:
Appendix F also needs more than Gherkin. The scenarios, the flag set they assume, and the control API that produces outages are three co-dependent artifacts — a feature file that evaluates
boolean-flagis meaningless without the flag definition, and a disconnect scenario is meaningless without the endpoint that produces the disconnect. An OpenAPI document has no home under "Gherkin Suites".Both appendices now cross-reference each other.
Contents
specification/appendix-f-provider-conformance.md— the appendix, written for implementers in other languages: what the artifacts are, the control-API contract and its two invariants, the capability tag vocabulary, the six things a language implementation owns, and the rules for extending the suitespecification/assets/provider-tck/gherkin/— four feature files (evaluation, errors, events, lifecycle)specification/assets/provider-tck/flags/canonical-flags.json— the flag set every scenario assumesspecification/assets/provider-tck/openapi/control-api.yaml— what a backend under test must exposespecification/assets/provider-tck/README.md— a landing page for the directory, matching the convention ofassets/gherkin/README.mdspecification/assets/provider-tck/go.modandembed.go— the directory is also a nested Go module exposing the artifacts as anembed.FS. A Go module zip carries a git submodule only as a gitlink, so the Go TCK depends on this module instead of vendoring a copy. The other languages build from a working tree and keep using the submodule; these two files are inert for them. Precedent:open-feature/flagd-schemas.Assets live under
assets/provider-tck/rather than alongside the existing SDK Gherkin, becauseassets/gherkin/evaluation.featurealready exists and means something different.A
.gitattributesnormalises these files to LF. They are consumed byte for byte by every language's TCK and copied verbatim into published build artifacts, so a Windows checkout must not produce a different packaged file than a Linux one.Design decisions worth reviewing
Capabilities. Not every provider implements every optional part of the contract. Scenarios exercising an optional capability carry a tag; a provider declares what it supports and the rest are reported as skipped with a reason, never as passed. The rule behind it: a conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all. Reserved tags (
@targeting,@caching) have no scenarios yet and must not be declarable, so an adopter declaring "everything" cannot pick up a vacuous claim.@lifecycleand@eventsare separate. A provider can perform an initialisation that reaches its backend without ever emitting events, and vice versa. An earlier draft conflated them, which forced providers to declare a capability they did not have in order to run scenarios they could satisfy.@numeric-coercionis genuinely optional. The specification does not require a provider to reportTYPE_MISMATCHwhen a float flag is requested as an integer; that rule comes from a flagd ADR and the tag borrows its name. An earlier draft of this PR claimed the opposite and called not declaring the tag "an admission of a known bug". That was wrong, and the appendix now says so. What remains true is that silently narrowing0.5to0is bad for users, and a provider withholding the capability should say whether that is a choice or a tracked defect. The report envelope in #425 hasknownDeviationsfor the second.No container restarts. Unavailability is simulated inside the running stack via the control API. Orchestrators assign host ports dynamically and cannot reliably preserve them across a restart, so restarting silently invalidates every provider pointed at the old port — and the failure looks like a flaky provider.
In-process control is a narrow carve-out. Providers with no backend (in-memory, environment-variable, file-based) may be driven in-process. Providers with a backend must use the control API; reaching an external backend through a side channel produces a suite that passes while proving nothing.
Extensions may add questions but never replace one. An adopter can ship its own feature files alongside the canonical set, but an extension file cannot occupy a canonical path, an extension scenario can never satisfy a canonical one, and a run that did not execute the canonical set in full must fail. The last rule was found by accident: a test selector matching one scenario produced a green suite and a well-formed report describing one scenario out of twenty-nine.
Open questions
GET /last-evaluationor similar). Until then a provider that silently drops the context passes. The@targetingtag is reserved for those scenarios.@numeric-coerciontests only the lossy half. A provider that wrongly rejects10.0as an integer passes. Closing that needs an integral float in the canonical flag set, which changes the flag set for every language at once. Accessor width (32-bit versus 64-bit integer accessors) is not modelled either.Implementations
tools/provider-tckin open-feature/java-sdk-contrib#1830, adopted by the flagd provider for both its RPC and in-process resolvers. It consumes these files from this repo via submodule, and the artifacts were verified to round-trip byte-identical.Which of #417's open questions this answers
Q1, directory layout — proposes
specification/assets/provider-tck/{gherkin,flags,openapi}/. The subdirectory is forced rather than chosen:assets/gherkin/evaluation.featurealready exists and tests an SDK, not a provider.Q4, numeric coercion as capability vs hard requirement — keeps it a capability, and the appendix now states plainly that the specification does not require the behaviour, so a provider that coerces is not violating the spec. The distinction between "chose not to" and "known defect, tracked here" is left to the report format in #425 rather than to the tag vocabulary.
Left open, unchanged by this PR: Q2 versioning, Q3 canonical flag-set format, Q5 context passthrough, Q6 reason values, Q7 conformance claims. Q5 is also recorded in the appendix's own "Open questions" section; Q7 is split out into #424.
What the suite has found so far
Two real defects, from the outside, which is the argument for the whole exercise:
MultiProviderswallows child provider events — it extendsEventProviderbut never subscribes to its children, soPROVIDER_CONFIGURATION_CHANGED,PROVIDER_ERRORandPROVIDER_STALEnever reach the client. Independently reproduced by running the suite against it; already tracked as open-feature/java-sdk#1882 (gap 1, High), where it had been found by hand-comparing against the js-sdk reference.