feat(provider-tck): identify a Scenario Outline row by its parameters

A scenario entry in the conformance report was identified by feature and name.
Every row of a Scenario Outline shares one name, so the type-mismatch matrix in
errors.feature produced eleven entries that differed only in durationMs. If one
row failed and ten passed, the report could not say which failed, and a consumer
keying on feature and name kept whichever row it read last.

Per the report schema, a scenarioResult now carries `example`: the Examples row
it came from, keyed by column header, with the cells verbatim as strings.
Gherkin has no types, so "1" stays "1" rather than becoming 1 -- the report says
what the table said. It is present only for outline rows and omitted otherwise.

godog hands a hook an already-expanded pickle, whose step text has the
parameters substituted in and whose row is otherwise gone. What survives is
AstNodeIds, whose last entry is the id of the Examples TableRow. The row is
therefore recovered by parsing the embedded feature files a second time and
indexing every TableRow by that id. Those ids come from a counter godog creates
once per run and shares across the files it parses, so reproducing them means
reproducing godog's parse -- same files, same order, pickle compilation in
between. That coupling is not left to be trusted: a pickle that came from an
outline and did not resolve fails the run, because quietly returning to the
ambiguity this field exists to remove is worse than a build failure.

The capability gate records its outcome before a scenario starts, so it fills
the field in too. Four skipped rows of the @object outline are as ambiguous as
four failed ones.

The gate's own bookkeeping is keyed by pickle id rather than by scenario name
for the same reason. Gherkin allows an Examples block to carry its own tags, so
two rows of one outline can differ in whether the gate stops them; keyed by
name, gating one row suppressed the after hook for every row and the rows that
did run would have vanished from the report.

gherkin/go/v26 moves from an indirect requirement to a direct one. It is the
same module and version godog already builds against, so no dependency is added
and no go.sum entry changes.

Verified against the schema on open-feature/spec#425 with a Draft 2020-12
validator: every report the self-tests emit validates, the eleven matrix rows
carry eleven distinct examples matching the feature file, and (feature, name,
example) is unique across every scenario in a report.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>

Author: aepfli

tools/provider-tck/README.md

@@ -260,6 +260,44 @@ artifacts report the same value even when pinned to different commits — and it
 *configuration*, and it is reported as such. One provider with two materially different modes
 produces two reports that are not interchangeable.
 
+### What identifies a scenario
+
+`feature` and `name` together do not. Every row of a Scenario Outline shares one name, and the
+type-mismatch matrix in `errors.feature` is eleven rows, so eleven entries carry the same feature and
+the same name. A report that stopped there could not say which row failed, and a consumer keying on
+the pair would keep whichever row it read last.
+
+A row is identified by its parameters, which the report carries in `example` — the Examples row it
+came from, keyed by column header:
+
+```console
+$ jq -c '.scenarios[] | select(.name | startswith("Requesting the wrong type")) | .example' reports/in-memory.json
+{"default":"false","key":"string-flag","requested":"Boolean"}
+{"default":"1","key":"string-flag","requested":"Integer"}
+{"default":"0.1","key":"string-flag","requested":"Float"}
+...
+```
+
+The values are the cells verbatim, as strings. Gherkin has no types, so `"1"` stays `"1"`: coercing
+it would be this implementation inventing a fact the feature file did not state, and four
+implementations would each invent a different one.
+
+It is a field rather than a naming convention because the parameters *are* the identity, and they
+come from the feature file rather than from any runner. Mandating a mangled name instead would put a
+separator, an ordering and an escaping rule into normative text that every implementation has to
+reproduce byte for byte, and drift there is invisible until two reports silently fail to line up.
+
+A skipped row carries it too. The capability gate records its outcome before the scenario starts, so
+the four rows of the `@object` outline would otherwise be four `not-declared` entries differing in
+nothing — exactly as ambiguous as four failures.
+
+godog's hooks receive an already-expanded scenario, whose step text has the parameters substituted
+into it and whose row is otherwise gone. What survives is `AstNodeIds`, whose last entry is the id of
+the Examples `TableRow`, so the row is recovered by parsing the embedded feature files a second time
+and indexing every row by that id. Those ids come from a counter godog shares across the files it
+parses, which means reproducing them means reproducing godog's parse; a run that cannot resolve a row
+it knows came from an outline fails rather than quietly emitting the ambiguity again.
+
 ## The self-tests
 
 Three suites run against providers from the SDK itself. They need no Docker and finish in

tools/provider-tck/go.mod

@@ -3,13 +3,13 @@ module github.com/open-feature/go-sdk-contrib/tools/provider-tck
 go 1.25.0
 
 require (
+	github.com/cucumber/gherkin/go/v26 v26.2.0
 	github.com/cucumber/godog v0.15.1
 	github.com/cucumber/messages/go/v21 v21.0.1
 	github.com/open-feature/go-sdk v1.18.0
 )
 
 require (
-	github.com/cucumber/gherkin/go/v26 v26.2.0 // indirect
 	github.com/gofrs/uuid v4.4.0+incompatible // indirect
 	github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
 	github.com/hashicorp/go-memdb v1.3.5 // indirect

tools/provider-tck/pkg/tck/examples.go

@@ -0,0 +1,231 @@
+package tck
+
+import (
+	"fmt"
+	"io/fs"
+	"sort"
+	"strings"
+	"sync"
+
+	"github.com/cucumber/gherkin/go/v26"
+	"github.com/cucumber/godog"
+	messages "github.com/cucumber/messages/go/v21"
+)
+
+// WHY THIS FILE EXISTS
+//
+// A row of a Scenario Outline is identified by its parameters. Every row shares
+// the outline's name, so a report that identifies a scenario by feature and name
+// gives eleven identical entries for the eleven rows of the type-mismatch matrix
+// in errors.feature. If one row fails and ten pass, that report cannot say which
+// failed, and a consumer keying on feature and name keeps whichever row it read
+// last.
+//
+// godog hands a hook a *godog.Scenario, which is a messages.Pickle. A pickle is
+// the already-expanded scenario: its step text has the parameters substituted
+// into it, and the row they came from is gone. What survives is AstNodeIds,
+// whose last entry, for a pickle compiled from an outline, is the id of the
+// Examples TableRow the pickle was expanded from.
+//
+// So the row is recovered by parsing the same feature files a second time and
+// indexing every Examples TableRow by that id.
+
+// exampleRow is one row of an Examples table.
+type exampleRow struct {
+	// values are the cells keyed by column header, verbatim as strings. Gherkin
+	// has no types, so "1" stays "1": coercing it to a number would be this
+	// implementation inventing a fact the feature file did not state, and the
+	// four language implementations would each invent a different one.
+	values map[string]string
+	// order is where the row sits among all the rows of its scenario, counting
+	// across every Examples block the scenario has. It is carried so the report
+	// can list the rows in the order the table declares them rather than in
+	// whatever order sorting the parameter values happens to produce.
+	order int
+}
+
+// exampleIndex resolves a pickle to the Examples row it was expanded from.
+type exampleIndex struct {
+	// rows is keyed by the id of the TableRow AST node.
+	rows map[string]exampleRow
+	// outlines names every scenario that is an outline, keyed by feature URI and
+	// scenario name. It exists so a failure to resolve a row can be told apart
+	// from a scenario that legitimately has none, and reported rather than
+	// silently emitting a report with the ambiguity this field exists to remove.
+	outlines map[string]bool
+}
+
+var (
+	exampleIndexOnce sync.Once
+	exampleIndexVal  *exampleIndex
+	exampleIndexErr  error
+)
+
+// scenarioExamples returns the index, building it once per process.
+func scenarioExamples() (*exampleIndex, error) {
+	exampleIndexOnce.Do(func() {
+		exampleIndexVal, exampleIndexErr = buildExampleIndex()
+	})
+	return exampleIndexVal, exampleIndexErr
+}
+
+// buildExampleIndex parses the embedded feature files and indexes their
+// Examples rows by AST node id.
+//
+// The id has to agree with the one godog will report, and godog's ids are not
+// intrinsic to a document: they come from a counter (messages.Incrementing) that
+// godog creates once per run and shares across every file it parses, so an id
+// depends on how many nodes were numbered before it. Reproducing them therefore
+// means reproducing godog's whole parse — the same files, in the same order,
+// with pickle compilation in between, because compiling pickles draws from the
+// same counter.
+//
+// That is a coupling to godog's internals, and it is a deliberate one: the
+// alternative is to fork the pickle compiler. It is not left to be trusted.
+// A pickle that comes from an outline and does not resolve is reported as a
+// failure by the run, so a godog release that renumbers nodes breaks the build
+// loudly instead of quietly emitting reports with the ambiguity removed again.
+func buildExampleIndex() (*exampleIndex, error) {
+	index := &exampleIndex{
+		rows:     map[string]exampleRow{},
+		outlines: map[string]bool{},
+	}
+
+	paths, err := featureFilePaths()
+	if err != nil {
+		return nil, err
+	}
+
+	newID := (&messages.Incrementing{}).NewId
+	for _, path := range paths {
+		file, err := assets.Open(path)
+		if err != nil {
+			return nil, fmt.Errorf("opening the embedded feature file %s: %w", path, err)
+		}
+		document, err := gherkin.ParseGherkinDocumentForLanguage(file, gherkin.DefaultDialect, newID)
+		closeErr := file.Close()
+		if err != nil {
+			return nil, fmt.Errorf("parsing the embedded feature file %s: %w", path, err)
+		}
+		if closeErr != nil {
+			return nil, fmt.Errorf("closing the embedded feature file %s: %w", path, closeErr)
+		}
+
+		document.Uri = path
+		index.collectDocument(document)
+
+		// The result is discarded; the side effect is the point. Compiling the
+		// pickles advances the id counter exactly as godog's parse advances it,
+		// so the next document's AST nodes are numbered the way godog numbers
+		// them.
+		_ = gherkin.Pickles(*document, path, newID)
+	}
+
+	return index, nil
+}
+
+// featureFilePaths lists the embedded feature files in the order godog walks
+// them, which is the lexical order fs.WalkDir yields.
+func featureFilePaths() ([]string, error) {
+	var paths []string
+	err := fs.WalkDir(assets, featuresPath, func(path string, entry fs.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		if entry.IsDir() || !strings.HasSuffix(path, ".feature") {
+			return nil
+		}
+		paths = append(paths, path)
+		return nil
+	})
+	if err != nil {
+		return nil, fmt.Errorf("listing the embedded feature files: %w", err)
+	}
+	// WalkDir already yields lexical order; sorting says so rather than relying
+	// on it, since the ids only line up if this order matches godog's.
+	sort.Strings(paths)
+	return paths, nil
+}
+
+func (index *exampleIndex) collectDocument(document *messages.GherkinDocument) {
+	if document == nil || document.Feature == nil {
+		return
+	}
+	for _, child := range document.Feature.Children {
+		if child == nil {
+			continue
+		}
+		if child.Scenario != nil {
+			index.collectScenario(document.Uri, child.Scenario)
+		}
+		if child.Rule == nil {
+			continue
+		}
+		for _, ruleChild := range child.Rule.Children {
+			if ruleChild != nil && ruleChild.Scenario != nil {
+				index.collectScenario(document.Uri, ruleChild.Scenario)
+			}
+		}
+	}
+}
+
+func (index *exampleIndex) collectScenario(uri string, scenario *messages.Scenario) {
+	if len(scenario.Examples) == 0 {
+		return
+	}
+	index.outlines[outlineKey(uri, scenario.Name)] = true
+
+	order := 0
+	for _, examples := range scenario.Examples {
+		if examples == nil || examples.TableHeader == nil {
+			continue
+		}
+		headers := make([]string, 0, len(examples.TableHeader.Cells))
+		for _, cell := range examples.TableHeader.Cells {
+			headers = append(headers, cell.Value)
+		}
+
+		for _, row := range examples.TableBody {
+			if row == nil {
+				continue
+			}
+			values := make(map[string]string, len(headers))
+			for i, cell := range row.Cells {
+				// A row with more cells than headers is malformed Gherkin the
+				// parser would have rejected; guarding costs nothing and keeps a
+				// future parser change from panicking here.
+				if i >= len(headers) {
+					break
+				}
+				values[headers[i]] = cell.Value
+			}
+			if len(values) > 0 {
+				index.rows[row.Id] = exampleRow{values: values, order: order}
+			}
+			order++
+		}
+	}
+}
+
+// rowFor resolves the Examples row a pickle was expanded from.
+//
+// The last AST node id is the TableRow for an outline pickle and the Scenario
+// node for an ordinary one, so a lookup that misses is the ordinary case rather
+// than an error.
+func (index *exampleIndex) rowFor(sc *godog.Scenario) (exampleRow, bool) {
+	if sc == nil || len(sc.AstNodeIds) == 0 {
+		return exampleRow{}, false
+	}
+	row, ok := index.rows[sc.AstNodeIds[len(sc.AstNodeIds)-1]]
+	return row, ok
+}
+
+// isOutline reports whether a scenario name in a feature belongs to a Scenario
+// Outline, and therefore must carry an example.
+func (index *exampleIndex) isOutline(uri, name string) bool {
+	return index.outlines[outlineKey(uri, name)]
+}
+
+func outlineKey(uri, name string) string {
+	return uri + "\n" + name
+}

tools/provider-tck/pkg/tck/report.go

@@ -90,22 +90,40 @@ type ReportCapability struct {
 }
 
 type ReportScenario struct {
-	Feature    string   `json:"feature"`
-	Name       string   `json:"name"`
-	Tags       []string `json:"tags,omitempty"`
-	Outcome    Outcome  `json:"outcome"`
-	Reason     string   `json:"reason,omitempty"`
-	DurationMs float64  `json:"durationMs,omitempty"`
+	Feature string `json:"feature"`
+	Name    string `json:"name"`
+	// Example is the Examples row this entry came from, keyed by column header,
+	// present only for a scenario expanded from a Scenario Outline.
+	//
+	// It is what makes such an entry identifiable. Feature and name are shared by
+	// every row of an outline -- eleven rows of the type-mismatch matrix in
+	// errors.feature produce eleven otherwise identical entries -- so without it
+	// a report cannot say which row failed.
+	//
+	// The values are the cells verbatim, as strings. Gherkin has no types, so the
+	// cell "1" is reported as "1" and not as 1; the report says what the table
+	// said and leaves the interpretation to whoever reads it.
+	Example    map[string]string `json:"example,omitempty"`
+	Tags       []string          `json:"tags,omitempty"`
+	Outcome    Outcome           `json:"outcome"`
+	Reason     string            `json:"reason,omitempty"`
+	DurationMs float64           `json:"durationMs,omitempty"`
 }
 
 // scenarioRecord is what the runner accumulates as scenarios execute.
 type scenarioRecord struct {
-	feature  string
-	name     string
-	tags     []string
-	outcome  Outcome
-	reason   string
-	duration time.Duration
+	feature string
+	name    string
+	example map[string]string
+	// exampleOrder is the row's position in its scenario's Examples tables, kept
+	// only to sort the report. Ordering by the parameter values would list the
+	// rows of a matrix in an order the feature file never mentions, which makes a
+	// report needlessly hard to read next to the table it came from.
+	exampleOrder int
+	tags         []string
+	outcome      Outcome
+	reason       string
+	duration     time.Duration
 }
 
 // buildReport assembles the report from what the run observed.
@@ -127,7 +145,10 @@ func (r *runner) buildReport() Report {
 		if records[i].feature != records[j].feature {
 			return records[i].feature < records[j].feature
 		}
-		return records[i].name < records[j].name
+		if records[i].name != records[j].name {
+			return records[i].name < records[j].name
+		}
+		return records[i].exampleOrder < records[j].exampleOrder
 	})
 
 	scenarios := make([]ReportScenario, 0, len(records))
@@ -145,6 +166,7 @@ func (r *runner) buildReport() Report {
 		scenarios = append(scenarios, ReportScenario{
 			Feature:    rec.feature,
 			Name:       rec.name,
+			Example:    rec.example,
 			Tags:       rec.tags,
 			Outcome:    rec.outcome,
 			Reason:     rec.reason,