test(flagsmith): run the provider conformance suite against the testbed

[aepfli]

Runs the OpenFeature Provider Conformance Suite against the Flagsmith JavaScript provider.

Stacked on feat/provider-tck. Companion to go-sdk-contrib#959 and java-sdk-contrib#1849 — same container, same scenarios.

Result: 31 pass, 15 fail, 19 skipped of 65

	Go	Python	Java	JS
passed	45	28	35	31
failed	4	5	13	15
skipped	16	23	17	19
of	65	56*	65	65

* Python's TCK has not picked up @standard-reasons yet.

@standard-reasons is what changed this run. It was 19 pass / 14 fail before that capability existed; ten of those fourteen were this provider's reason handling, and they are now skips carrying a recorded reason.

@standard-reasons — declared, and failing

The provider reports the reason as flag.enabled ? TARGETING_MATCH : DISABLED — from the enabled state alone, with no targeting involved and without consulting the evaluation context at all. So every enabled flag claims a targeting rule matched, when the canonical set carries exactly one rule and the flag in question has none.

2.2.5 makes the reason a SHOULD, which is why this is a withheld claim rather than a failure. But unlike Java's null, TARGETING_MATCH for an untargeted flag is not defensible on SHOULD grounds — it is a wrong answer rather than a missing one. Go reports STATIC, DISABLED and TARGETING_MATCH correctly against the identical backend.

The 4 remaining failures — a silent wrong-type success

boolean-flag, integer-flag, float-flag and object-flag requested as a String all return their value stringified — "true", "10", "0.5", the raw JSON — rather than TYPE_MISMATCH.

The two halves are worth separating:

• float and object fail in every language, because Flagsmith really does store both as strings, so the request is not a type mismatch on this backend.
• boolean and integer are genuine type mismatches that only this provider gets wrong. An application asking for a boolean flag as a string gets "true" and no error, which is the worst failure mode a flag has: a plausible value and no signal.

@numeric-coercion — no deviation here, deliberately

The TCK refused one, and it was right to. JavaScript has a single numeric type, so "a float requested as an integer" is not expressible and nothing was ever put to this provider. Recording a deviation would have claimed a failure at something never asked.

Go, Java and Python all keep theirs, because in those languages the question can be asked and the answer is wrong. Worth noting because an earlier version of this description said this provider "behaves like Python and Java" on floats — it does not; in JavaScript the distinction does not exist.

@disabled-flags — declared, and failing

A disabled flag raises GeneralError rather than resolving to the caller default with no error code. Neither configuration satisfies the scenario: returnValueForDisabledFlags defaults false and throws, and setting it true returns the flag's configured value instead of the caller default — which the scenario also catches, because each disabled flag's configured value differs from the default the scenario passes in.

Go and Java both return the caller default with reason DISABLED and declare the capability. Python has the same defect as this one.

useBooleanConfigValue — the four-way split

Set true here so the suite tests the same semantics as the other adoptions. Its default is false, meaning a boolean flag resolves from Flagsmith's enabled state rather than feature_state_value. That matches Python's default and is the opposite of Go and Java — four providers for one product, split two-two on what a boolean flag is.

Other capabilities

Declared: @object, @large-integers, @targeting.

@variants is withheld because Flagsmith has no variant concept for a plain feature — permitted rather than defective, so no deviation. The lifecycle and event capabilities are withheld because this provider has no observable initialisation.

Workspace changes worth review

• libs/providers/flagsmith/tsconfig.json did not extend tsconfig.base.json, unlike every other provider, so the @openfeature/tck path mapping was invisible to it.
• The suite lives in src/tck behind a tck target, not src/e2e behind an e2e one. Both mistakes were made here first: the unit-test config picked the suite up by directory alone and broke 36 passing tests, and npm run e2e is a CI job, so a containerised suite the policy deliberately excludes was running there. This follows the documented npx nx tck providers-flagsmith convention.
• The e2e jest config omits flagd's generic '(.+)\.js$' -> '$1' mapper. It is too greedy: it rewrites package names ending in -js and breaks @grpc/grpc-js. Worth checking whether flagd's copy has the same latent problem.
• No testTimeout. runProviderTck calls jest.setTimeout itself now, derived from the suite's own timeouts, so a value here would read as the bound while a different one was in force.

The backend

aepfli/flagsmith-tck-testbed 0.1.0, pinned rather than :latest because four adoptions pull it and a mutable tag lets a push change four PRs' results with no diff to explain it.

Why this is a draft

• The testbed image lives in a personal namespace.
• Only libs/providers/flagsmith is covered; flagsmith-client is untouched.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}