feat(provider-tck): add the Go conformance suite for OpenFeature providers

[aepfli]

Closes #938
Part of open-feature/spec#417. The
language-agnostic artifacts are in spec#423; the report envelope in
spec#425.

Draft — opened for review of the approach. The module builds, golangci-lint is clean, and
the self-test suites pass with no Docker. The open questions are where I would
most value pushback.

The PR chain

The same four-branch shape in every language, so understanding one is understanding all four:

main
 └── #940  feat/provider-tck            this PR — the suite, both control paths, the compose
      │                                  harness, the extension point, and the entire
      │                                  adopter-facing API
      ├── #944  feat/provider-tck-report    Cucumber Messages and the report envelope
      └── #941  feat/provider-tck-flagd     flagd adoption, both resolvers
           └── #942  feat/provider-tck-ofrep   OFREP adoption

report and flagd are siblings, not a sequence: the follow-ups fill gaps without changing the
adopter-facing API, so neither blocks the other and either can land first.

What this is

A conformance suite any Go provider can adopt to check that it implements the provider contract in
the specification — the Go implementation of Appendix F, running the same Gherkin
scenarios, against the same canonical flag set, driven through the same control API as every other
language's TCK. That shared basis is the point: "conformant" only means something if the question is
identical everywhere.

Java is the reference (java-sdk-contrib#1830 and its stack). This is not a transcription of
it — Go has no inheritance, no JUnit Platform Suite and no exceptions, and each of those forced a
decision rather than a translation.

The adoption surface

tck.Run with functional options. Three are required — a name, a provider factory, and the seam
through which the backend is driven:

func TestMyProviderConformance(t *testing.T) {
	tck.Run(t,
		tck.WithName("my-provider"),
		tck.WithComposeFile("testdata/docker-compose.yaml"),
		tck.WithBackendPorts(8013),
		tck.WithProviderFromEndpoint(func(ctx context.Context, e tck.BackendEndpoint) (openfeature.FeatureProvider, error) {
			return myprovider.New(e.Host(), e.Port(8013)), nil
		}),
		tck.WithCapabilities(tck.Events, tck.Object, tck.Variants),
	)
}

Options rather than a Config struct, at @erka's request:
it leaves room for a tck that tests something other than a provider later, instead of a second
package duplicating the harness. Run validates up front and names what is missing.

The suite owns the container stack. WithComposeFile plus WithBackendPorts is the whole of it
— starting Compose, discovering dynamically mapped host ports, building the control client, waiting
for the control API, tearing down. Conventions with working defaults: WithBackendService
(backend), WithControlPort (8080), WithAdditionalPorts, WithBackendConfiguration
(default), WithStartupTimeout (60s). This is now required by Appendix F, after three of four
languages originally shipped only the control-API client and left orchestration to the adopter — at
which point every adoption hand-rolled between 130 and 460 lines of the same wrapper.

A provider with no backend passes tck.WithControl and its own BackendControl instead, and
supplies tck.WithProvider — a factory taking no endpoint. Two factory options rather than one
because Go cannot overload, and a single signature would hand backend-less adopters a zero-valued
endpoint that means nothing.

Each scenario becomes a Go subtest, so -run selects one and a failure names a scenario. The
canonical Gherkin and flag set arrive through the spec's own nested Go module, so adopters need no
submodule and no vendored copy — a copy would be a second place for conformance to drift.

Declaring what a provider can and cannot do

WithCapabilities names which optional parts of the contract this provider supports, from fourteen
tags — tck.Events, tck.Lifecycle, tck.Stale, tck.Targeting, tck.DisabledFlags,
tck.NumericCoercion, tck.LargeIntegers, tck.Reinitialization, tck.ConfigurationChange,
tck.Object, tck.Variants, tck.UnavailableInit, tck.StandardReasons, and tck.Caching,
which is reserved and
cannot be declared. A scenario gated on an undeclared capability is reported skipped with its
reason, never passed.

WithKnownDeviations is the separate statement that the provider fails something it is required to
do. Appendix F settles which of its two shapes to prefer: declare the capability, let the scenario
fail, and record the deviation beside the failure — rather than withdrawing the capability so the
scenario skips, which hides a defect behind something that looks deliberate.

reason is asserted in two places only, and they mean different things. Requirement 2.2.5 is a
SHOULD permitting "some other string", so the scenarios that resolve a value no longer assert a
reason at all — an exact-match assertion there fails providers the specification permits. What
replaced it is a capability. tck.StandardReasons declares that this provider speaks the
specification's own vocabulary, and a feature file then checks the whole of it: STATIC,
TARGETING_MATCH, DEFAULT, DISABLED. Undeclared, those scenarios skip with their reason, and
nothing else in the suite cares what a provider calls a reason.

The error cases are not part of that bargain and are asserted unconditionally. Requirement 1.4.8
makes an error code mandatory in abnormal execution, so ERROR there is not a dialect — a reason
that disagrees with the error code shipped beside it is a contradiction. The assertion is on the two
agreeing, not on which of them is authoritative.

Three Go-specific decisions

1. "no exception should have been thrown" asserts the evaluation did not panic. Go has no
exceptions, and the returned error is not one — an errored evaluation correctly returns the code
default alongside a non-nil error, which is the normal shape of the API. What the feature files
forbid is an unhandled failure escaping a flag evaluation and taking the host application down,
which here is a panic. Provider registration is guarded the same way.

2. Providers are registered under a suite-scoped domain, not a per-scenario one. Registering in
a domain replaces and shuts down the previous provider. A fresh domain per scenario reads cleaner
but would leave every provider of the suite registered and running — for a provider holding a
network connection, one leaked connection per scenario.

3. Capability gating is godog.ErrSkip returned from the Before hook. godog treats that as
"skip this scenario and all its steps" rather than a failure, and errors.Is still matches after
godog wraps it. Every skip is collected and printed with its reason at the end of the suite, because
a suite that quietly goes green on scenarios it did not run is worse than no suite at all.

A non-obvious constraint the design depends on: the SDK compares providers with reflect.DeepEqual
unless the dynamic type is a pointer. A value-typed provider holding an equal flag set would compare
as the same provider, and the per-scenario replacement the TCK relies on for isolation would
silently not happen. ControllableProvider is therefore pointer-only, and says so.

Self-tests: three suites, no Docker

Suite	Subject	Why
TestInMemoryProvider	memprovider.InMemoryProvider	reference adoption for a backend-less provider
TestControllableProvider	tck.ControllableProvider	the only suite exercising the configuration-change and lifecycle paths
TestMultiProvider	multi.Provider wrapping one child	delegation must be transparent

TestMultiProvider wraps exactly one child deliberately. That is the interesting configuration
rather than a degenerate one: the correct answer is precisely what TestControllableProvider
already asserts about the child alone, so any difference between the two suites is attributable to
the multi-provider and nothing else — a variant that does not survive the hop, a reason rewritten to
DEFAULT, an error code flattened to GENERAL, an event that never reaches the client.

Two findings against the Go SDK

The in-memory provider cannot update its flag set. Appendix A is unambiguous:

The provider must support a means of updating the flag set, resulting in the emission of
PROVIDER_CONFIGURATION_CHANGED events.

memprovider.InMemoryProvider has no update method, is not an EventHandler, and is not a
StateHandler. JavaScript has putConfiguration() and Java has updateFlag(); Go has neither. The
knock-on is not confined to this suite — Appendix A also requires SDK end-to-end tests to use the
in-memory provider, so go-sdk's own Appendix B suite cannot cover configuration-change
events either. Tracked as go-sdk#530.

Handled honestly rather than worked around: TestInMemoryProvider leaves ConfigurationChange
undeclared so the scenario is skipped with the reason, and ControllableProvider supplies the
missing behaviour by wrapping the SDK's provider rather than reimplementing it — every
resolution decision is still memprovider's — so it doubles as a reference for what the SDK's
provider should grow.

A disabled flag returns a GENERAL error alongside the default. in_memory_provider.go returns
defaultValue with DisabledReason and NewGeneralResolutionError, which contradicts
requirement 2.2.6. Already filed as go-sdk#552
and fixed by #574, but no release carries the fix
yet — so this module still pins v1.18.0 and TestInMemoryProvider withholds @disabled-flags. The
pin and the withheld capability should both be revisited once a release lands.

Running it

The self-test suites run in the default build with no Docker. The adoption suites are gated:

The gating half of this is provisional and under discussion. The suite is excluded because its
honest output is red — but that is only true while a run is judged by "zero failures". Judged
instead by whether its results match its declaration — every failure covered by a declared
deviation, every skip gated by an undeclared capability, every deviation still failing something —
a healthy adoption is green in its steady state, deviations included, and this could be a required
gate. Proposed in
open-feature/spec#417;
nothing here depends on the outcome. What is settled either way is that the conformance suite gets
a step of its own rather than being folded into the provider's e2e suite, so a red build says
conformance failed rather than a test failed.

make tck

Appendix F now carries the reasoning: an adoption suite's honest output is red while real gaps
remain — filed provider defects, missing backend fixtures — so making it a required gate forces
someone to silence it, and the cheapest way to silence a conformance suite is to stop asking the
question.

Two mechanisms, doing two different jobs. Neither substitutes for the other, and saying which is
which is the whole of the answer to
@erka's question about a build tag:

• The module path decides which target runs a suite. An adoption is a module of its own at
  providers/<name>/tck, beside the provider's e2e suite rather than inside it. make tck runs those
  modules; make e2e runs the others. A file is in the directory or it is not.
• A //go:build tck tag decides nothing about that. It keeps the suite out of every invocation
  that asks for no tags at all — make test, and a bare go test ./... typed in the module — which
  would otherwise start a Docker stack.

Why a build tag is safe here when the objection to one was that it is not. The objection is that
a tag takes the suite out of compilation, and Appendix F asks for the opposite: the suite must keep
compiling even when it does not execute, so a signature change in the harness cannot rot an adoption
unnoticed. That holds only while nothing in the pipeline builds with the tag. Something now does:
make e2e's second command builds the conformance modules under -tags=tck with an empty -run
pattern, which keeps the build and drops the run. Verified, not asserted:

ok  github.com/open-feature/go-sdk-contrib/providers/flagd/tck  0.175s [no tests to run]
ok  github.com/open-feature/go-sdk-contrib/providers/ofrep/tck  0.161s [no tests to run]

So the honest answer to "why not a build tag?" is neither "a tag is wrong" nor "an env var is
better". The question had two halves — an exclusion, and a named thing to run — and only the first
had an answer at the time. make tck is the second half, and with it in place the tag is right.

Two earlier mechanisms are gone. A TCK_RUN environment variable, because an exclusion hidden
inside a test function is invisible from the build and is not the step the appendix asks for. And a
-run 'Conformance' / -skip 'Conformance' test-name split, because it made the name of a test
load-bearing and needed a guard in every adoption to hold the correspondence. What each adoption
still carries is the other half of that guard: a test that fails if nothing in the module reaches
tck.Run, since a conformance module whose tests have stopped running the suite leaves make tck
green by running nothing. It is untagged, so it runs in the build the suite is absent from and reads
the tagged file off disk.

Worth stating plainly: this PR previously described the adoptions as excluded when make e2e was in
fact running them, red, on every PR. All four languages had a version of that, each defeated by a
different mechanism.

Verification

At abe0f9a6:

Check	Result
go build ./..., go vet ./..., go test ./... in the module	pass, no Docker
golangci-lint run from inside the module	exit 0 — note any exit code other than 0 or 1 means it typechecked nothing and did not actually run
Three self-test suites	green
Assets consumed from the spec's nested module	yes — pinned by pseudo-version in every go.mod, so no copy exists to drift
Adoption suites, run deliberately	flagd RPC and in-process both reach their recorded tally; every failure is a documented backend fixture gap

Known gaps

• Evaluation context passthrough beyond the targeting key. targeting-key-flag resolves
  differently for a matching context, so a provider that drops the context entirely is caught by the
  resolved value. What is still unverified is that the whole context arrives: a provider
  forwarding the targeting key and silently discarding every other attribute passes. Closing it
  needs an echo operation on the control API, or a canonical flag whose rule keys on a custom
  attribute.
• Targeting and bucketing correctness — out of scope by design; that is backend evaluation logic.
• Caching — @caching is reserved, no scenarios yet. Note for whoever writes them: flagd's RPC
  resolver runs an LRU cache by default and reports CACHED on a repeat evaluation, so any scenario
  evaluating the same flag twice sees a different reason the second time from a correct provider.
• Per-flag set/remove control operations — the control API defines reset-to-baseline and mutate
  one designated flag; finer-grained operations would need endpoints that do not exist.
• Hooks — not covered.

Open questions

1. Does the compose harness belong in this module? It puts testcontainers and the Compose
   client in tools/tck, so a backend-less adopter carries the go.sum entries and compiles
   nothing it does not import. The alternative is a second module with BackendEndpoint visible to
   both. Java made it provided/optional, Python an extra, JS an optional peer — four answers,
   each written down, none obviously wrong.
2. Nothing validates an emitted report against the schema in CI, in any language. The schema is
   what makes reports comparable, so this is the weakest link in the design, and it wants one answer
   rather than four.
3. Should ControllableProvider ship here, or should the fix land in go-sdk first and this
   module depend on it?

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[erka]

Hey @aepfli

Could we rename provider-tck to just tck? It isn't a provider, and the current name is a bit confusing.

Also, could we make it a flat package instead of putting it under pkg? pkg isn't really common practice anymore.

[aepfli]

Hey @aepfli

Could we rename provider-tck to just tck? It isn't a provider, and the current name is a bit confusing.

Also, could we make it a flat package instead of putting it under pkg? pkg isn't really common practice anymore.

Hey,

Sure thing, this is just me drafting and testing with Claude code, for multiple languages, and I try to stick to how the repo is handling things.

The naming is another question. What if there is a tck for other areas (hooks, listeners - bad examples as they are just interfaces currently) should we be ready for this?

[erka]

The naming is another question. What if there is a tck for other areas (hooks, listeners - bad examples as they are just interfaces currently) should we be ready for this?

It really depends on the implementation. If tck.Run(t, ....) has options instead of one struct it may give freedom to run suites for different areas.

[aepfli]

Both make sense — tools/tck, flat, no pkg. The import path stutters today as tools/provider-tck/pkg/tck.

On options vs a struct: agreed, and it lines up with a request from @toddbaert on the spec PR — adopters should be able to add their own scenarios and step definitions that run in the same backend phase, rather than maintaining a parallel harness for provider-specific features. Go has no runtime scanning, so that needs a registration hook:

tck.Run(t,
    tck.WithProvider(newProvider),
    tck.WithControl(control),
    tck.WithCapabilities(tck.Events, tck.Object),
    tck.WithSteps(func(ctx *godog.ScenarioContext) { /* vendor steps */ }),
    tck.WithFeatures(os.DirFS("testdata/tck-extensions")),
)

I'd do the rename and the options change in one go so the API only churns once. Happy to fold both into this PR — or split the rename out first if you'd rather review them separately.

[erka]

It looks great. Is there any reason to use TCK_RUN env instead of a new new go build tag? @aepfli

[aepfli]

It looks great. Is there any reason to use TCK_RUN env instead of a new new go build tag? @aepfli

to be honest, I have not thought about dedicated execution, and if this should run under the e2e build tag, an own build tag, or in general. The env var is currently just a small first draft version. But you got me thinking, and most likely we even want to have an own build step for this. But this is definitely worth a discussion, i have so many findings, and currently the approach is highly opinionated.

What would be your ideal approach? @erka

[erka]

What would be your ideal approach?

I see that using env vars is a different approach compared to previous e2e tests. I'm just curious about it and don't have any strong ideas. @aepfli