You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
v2/conformance: add POSIX/GNU suite + divergence oracle
Add the second conformance promise (POSIX/GNU argument syntax) alongside
the vendored stdlib flag suite, plus the machinery to reconcile the two
when they conflict.
- gnuposix_test.go: hand-written suite keyed to the POSIX Utility Syntax
Guidelines (1-14) and the GNU long-option extensions. No reusable
Go-native POSIX corpus exists to vendor, so each case cites the rule it
checks.
- divergences.json: single source of truth cataloguing the intentional
differences between pflag and the stdlib flag package. POSIX wins, so
the listed stdlib tests are *expected* to fail; each entry names them
and the rule that overrides them. Categories: posix-overrides-stdlib,
pflag-design-differs, pflag-omits-gnu-feature, not-implemented-yet.
- hack/oracle: reads `go test -json` and gates CI on the catalogue rather
than raw pass/fail: green iff failures match the documented set exactly
(catches regressions and silently-resolved divergences). The top-level
"status" field tolerates a build failure while "bootstrapping" so CI is
green during v2 build-out; "not-implemented-yet" entries are lenient
(tolerated when failing, retirable when they start passing).
- internal/divergence: catalogue types + parsing, shared by the manifest
test and the oracle (no build tag, no v2 dependency).
CI conformance-v2 job now pipes the suite through the oracle and drops
continue-on-error; it is green now (bootstrapping) and tightens to
enforcing once v2 compiles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| v2 compiles, partial |`enforcing`| add `not-implemented-yet` entries for tests that fail only because a feature is missing | build required; those failures tolerated → **green**|
56
+
| v2 complete |`enforcing`|`not-implemented-yet` burned down to empty | only permanent divergences remain |
57
+
58
+
`not-implemented-yet` is lenient: a listed test may fail, be skipped, or not run.
59
+
When one starts **passing**, the oracle stays green and just lists it as
60
+
*retirable* — delete the entry. (A *permanent* divergence that passes is the
61
+
opposite: a hard failure, because it never should.) So implementing a feature
62
+
never turns CI red; forgetting to categorise a new failure does.
|`doc.go`| hand-written | package doc; keeps the dir buildable with no tags; holds the `//go:generate` directive |
66
123
|`flag_go<ver>_test.go`| generated from that version's `flag_test.go`|**do not edit** — regenerate with `sync.sh`|
67
124
|`harness_test.go`| hand-written | reimplements the stdlib's internal `export_test.go` helpers (`ResetForTesting`, `DefaultUsage`) against v2's public API, plus the version guard |
125
+
|`gnuposix_test.go`| hand-written | the POSIX/GNU argument-syntax suite |
126
+
|`divergences.json`| hand-written | the catalogue of intentional differences (single source of truth) |
127
+
|`divergences.go` / `divergences_test.go`| hand-written | embed + validate the catalogue (untagged, runs in normal `go test`) |
128
+
|`internal/divergence/`| hand-written | catalogue types + parsing, shared by the test and the oracle (no build tag, no v2 dependency) |
129
+
|`hack/oracle/`| hand-written | the CI gate: compares `go test -json` against the catalogue |
68
130
|`internal/testenv/testenv.go`| hand-written | minimal stand-in for the stdlib-internal `internal/testenv` (only `MustHaveExec` / `Executable`) |
69
131
|`hack/sync.sh`| hand-written | regenerates the vendored copies |
70
132
@@ -79,6 +141,34 @@ edits, which the script applies and nothing else:
79
141
- inject a one-line `init()` after the imports registering the version with the guard
80
142
- prepend the build tags + a "generated" banner
81
143
144
+
# POSIX/GNU argument syntax suite
145
+
146
+
`gnuposix_test.go` checks pflag's *other* promise: compatibility with the POSIX
147
+
Utility Syntax Guidelines and the GNU long-option extensions. Each test cites the
148
+
rule it covers — `Gn` for a numbered POSIX Guideline, or "GNU" for an extension:
149
+
150
+
| Area | Rules covered |
151
+
| --- | --- |
152
+
| Short options | preceded by `-` (G4), single alphanumeric name (G3), clustering (G5), separate or attached arg (G6), cluster ending in an arg-taking option (G5) |
153
+
| Long options (GNU) |`--name`, dashes in names, `--name=value`, `--name value`|
154
+
| Special tokens |`--` ends options (G10), lone `-` is an operand (G13) |
155
+
| Ordering | GNU interspersing (default) vs strict POSIX stop-at-first-operand (G9), order-independence and repetition (G11) |
156
+
157
+
### Why hand-written instead of vendored?
158
+
159
+
Unlike the stdlib `flag` suite, there is **no reusable Go-native POSIX
160
+
conformance corpus to vendor**. The authoritative sources are prose — the
161
+
[Open Group Utility Conventions ch. 12][posix] (Guidelines 1–14) and
162
+
[GNU Argument Syntax][gnu] — and the machine-runnable suites (glibc/gnulib
163
+
`tst-getopt*.c`) are C, coupled to the C `getopt`/optstring API, so they mostly
164
+
exercise C-isms pflag does not share. The tests are therefore authored directly
165
+
from the guidelines. They are shaped as *parse → normalized result*, so if a
166
+
shared cross-language corpus ever appears we can expose a tiny CLI to drive it
"_comment": "Catalogue of intentional differences between pflag v2 and the standard library flag package. Single source of truth for both the human-readable docs and the conformance oracle (hack/oracle). Each entry lists the conformance tests it is expected to make fail. POSIX wins. This catalogue is a best-effort seed and MUST be calibrated against the real v2 implementation: the oracle reports any undocumented failure (a regression) and any permanent divergence that unexpectedly passes (to remove). Categories: posix-overrides-stdlib | pflag-design-differs | pflag-omits-gnu-feature | not-implemented-yet (temporary, burned down as v2 is built). The 'status' field gates build failures: 'bootstrapping' tolerates the suite not building yet (CI green during build-out); flip to 'enforcing' once v2 compiles.",
3
+
"status": "bootstrapping",
4
+
"divergences": [
5
+
{
6
+
"category": "posix-overrides-stdlib",
7
+
"topic": "single-dash tokens are short-option clusters; long names need the GNU \"--\" prefix",
8
+
"stdlib": "A single dash introduces a (possibly multi-character) flag name: \"-bool\", \"-int 22\", \"-help\".",
9
+
"pflag": "A single dash introduces one or more single-character options, so \"-int\" is the cluster -i -n -t; multi-character names require \"--\".",
10
+
"refs": "POSIX Guidelines 3, 4, 5, 14; GNU long-option extension",
11
+
"affectedTests": [
12
+
"TestParse",
13
+
"TestFlagSetParse",
14
+
"TestUserDefined",
15
+
"TestUserDefinedFunc",
16
+
"TestUserDefinedBool",
17
+
"TestHelp",
18
+
"TestExitCode"
19
+
]
20
+
},
21
+
{
22
+
"category": "pflag-design-differs",
23
+
"topic": "usage / PrintDefaults output uses the \"-s, --long\" layout",
24
+
"stdlib": "PrintDefaults emits \" -A\\tfor ...\" (single dash, tab-aligned) and the tests compare it byte-for-byte.",
25
+
"pflag": "Help reflects POSIX/GNU spelling (short and long forms together); the exact byte layout differs from the stdlib.",
26
+
"refs": "GNU usage convention",
27
+
"affectedTests": [
28
+
"TestPrintDefaults",
29
+
"TestUserDefinedBoolUsage"
30
+
]
31
+
},
32
+
{
33
+
"category": "pflag-design-differs",
34
+
"topic": "parse-error and usage message text differs from the stdlib",
35
+
"stdlib": "Errors read e.g. \"flag provided but not defined: -i\" and \"invalid value ... parse error\".",
36
+
"pflag": "pflag has its own error wording, so byte/substring comparisons against the stdlib text do not hold.",
37
+
"refs": "implementation detail (not mandated by POSIX)",
38
+
"affectedTests": [
39
+
"TestUsageOutput",
40
+
"TestParseError"
41
+
]
42
+
},
43
+
{
44
+
"category": "pflag-design-differs",
45
+
"topic": "no \"flag set before being defined\" panic",
46
+
"stdlib": "Defining a flag after Set was called for that name panics with \"flag X set at ... before being defined\".",
47
+
"pflag": "pflag does not track set-before-define; Set on an unknown flag returns an error and defining later does not panic.",
48
+
"refs": "implementation detail (not mandated by POSIX)",
49
+
"affectedTests": [
50
+
"TestDefineAfterSet"
51
+
]
52
+
},
53
+
{
54
+
"category": "pflag-design-differs",
55
+
"topic": "flag-name validation panics with different messages",
56
+
"stdlib": "Var panics with \"flag \\\"-foo\\\" begins with -\" / \"flag \\\"foo=bar\\\" contains =\".",
57
+
"pflag": "pflag validates names differently and does not produce the same panic messages.",
58
+
"refs": "implementation detail (not mandated by POSIX)",
59
+
"affectedTests": [
60
+
"TestInvalidFlags"
61
+
]
62
+
},
63
+
{
64
+
"category": "pflag-omits-gnu-feature",
65
+
"topic": "unambiguous long-option abbreviation",
66
+
"stdlib": "n/a",
67
+
"pflag": "GNU accepts \"--verb\" for \"--verbose\" when unambiguous; pflag requires the full long name.",
0 commit comments