feat: verify the release build in CI — build each package as-if releasing and assert wheel metadata

[scottschreckengaust]

Summary

Nothing in CI builds a package the way it is actually released, so defects introduced by the release-time transformation are only discoverable after publication. This proposes adding one step to the existing per-package matrix in .github/workflows/python.yml that builds each package as-if releasing and asserts the resulting wheel metadata is well-formed.

The gap

python.yml builds every changed package with uv build, but against the unpinned pyproject.toml:

https://github.com/awslabs/mcp/blob/main/.github/workflows/python.yml — Build package → uv build

The release path is different. release.yml mutates pyproject.toml first, then builds:

- name: Pin dependencies from lockfile      # only exists on the release path
- name: Build package
  run: uv build

That pin step runs only in release.yml, which is gated on tag + environment approval. Its first real execution is the one that publishes to PyPI. There is no job anywhere that runs pin-dependencies followed by uv build and looks at the result.

This is not hypothetical. Review of #2856 found four defects in the pin implementation — dev dependencies published as runtime deps, dropped environment markers making the package uninstallable off-platform, multi-version lock entries collapsing to a single wrong version, and failures exiting 0 — and every one was found by a reviewer manually running the command against a real package and installing the output, because CI could not.

Separately, #4458 documents four packages shipping test tooling as runtime dependencies on PyPI today. That is a different root cause (source misdeclaration, not the pin), but it is the same class of defect, it is visible in exactly the same place — the built wheel's Requires-Dist — and it has been live through many releases without CI noticing.

Proposal

Add one step at the end of the existing build job in python.yml. No new jobs, no new matrix, no secrets:

      - name: Build package as a release (pin, then build) and verify metadata
        working-directory: src/${{ matrix.package }}
        run: |
          set -euo pipefail
          if [ ! -f "uv.lock" ]; then
            echo "::debug::No uv.lock, release build not applicable"
            exit 0
          fi
          uv run --script ../../.github/workflows/release.py pin-dependencies --directory="."
          rm -rf dist
          uv build
          python3 - <<'PY'
          import glob, zipfile, sys

          whl = glob.glob('dist/*.whl')
          assert whl, 'no wheel produced'
          with zipfile.ZipFile(whl[0]) as z:
              meta = next(n for n in z.namelist() if n.endswith('.dist-info/METADATA'))
              lines = z.read(meta).decode('utf-8', 'replace').splitlines()

          reqs = [l.split(':', 1)[1].strip() for l in lines if l.startswith('Requires-Dist:')]
          # extras are opt-in (`pip install pkg[dev]`) and legitimately unpinned
          runtime = [r for r in reqs if 'extra ==' not in r]

          problems = []
          if not runtime:
              problems.append('no runtime requirements at all -- pin step produced nothing')
          unpinned = [r for r in runtime if '==' not in r]
          if unpinned:
              problems.append(f'unpinned runtime requirements: {unpinned}')

          if problems:
              for p in problems:
                  print(f'::error::{p}')
              sys.exit(1)
          print(f'release wheel OK: {len(runtime)} runtime requirements, all pinned')
          PY

Why this placement

• Must run last. Pinning leaves pyproject.toml intentionally inconsistent with uv.lock. uv sync --frozen tolerates that (exit 0) but uv sync --locked does not (exit 1, "lockfile needs to be updated"). Nothing in the repo uses --locked today (0 of 47 Dockerfiles, 0 workflows), but putting this step last means a future --locked step can't be broken by it.
• Mutates only the runner's checkout. pin-dependencies rewrites pyproject.toml in place; nothing is committed, and the job ends immediately after.
• Reuses the existing matrix, so per-package cost is one uv build plus a zipfile read.

What it would catch

defect	caught?	how
dev deps published as runtime (#4458, and finding 1 of #2856)	yes	with the optional dev-tool assertion below
dropped environment markers (finding 3 of #2856)	yes	the package became uninstallable; markers are visible in Requires-Dist
unpinned requirement slipping through	yes	'==' not in r
pin step silently producing nothing	yes	empty-runtime assertion
pin step exiting 0 on failure (finding 4 of #2856)	yes	set -euo pipefail + the assertion downstream
multi-version fork collapse (finding 2 of #2856)	partially	wheel is still well-formed; needs a lock-vs-metadata cross-check to catch fully

Optional second assertion (depends on #4458)

Adding a dev-tooling check would catch #4458's class directly:

DEV_PREFIXES = ('pytest', 'ruff', 'pyright', 'pre-commit', 'commitizen', 'coverage')
leaked = [r for r in runtime
          if r.split(';')[0].strip().lower().startswith(DEV_PREFIXES)]
if leaked:
    problems.append(f'test/lint tooling as runtime requirement: {leaked}')

This would fail today on the four packages in #4458, so it should land after that issue is fixed — otherwise it needs a temporary allow-list, which tends to outlive its usefulness.

One caveat learned the hard way while gathering the #4458 data: a name-prefix blocklist produces false positives. argcomplete and termcolor initially looked like leaks in ccapi-mcp-server, but uv tree --invert --package argcomplete shows argcomplete → checkov → awslabs-ccapi-mcp-server — they are legitimate transitive deps of checkov, a real runtime dependency. Keep the prefix list narrow (test runners and linters only, not their transitive closures), or check provenance rather than names.

Prior verification

The approach is already known to work across the fleet. Before filing this I ran exactly this sequence — pin-dependencies → uv build → read wheel METADATA — against all 59 Python packages under src/:

• 59/59 pinned and built successfully; 0 pin failures, 0 build failures
• 0 unpinned unconditional Requires-Dist entries
• environment markers preserved (7–27 per package)
• built wheel installs into a clean venv and imports (spot-checked on eks-mcp-server)
• pinning twice is idempotent (identical checksum; second run is a no-op)

So enabling this step should be green on main from day one, with the single exception of the optional dev-tooling assertion above.

Full data: #2856 (comment)

Note on container images

Worth recording, since it is easy to assume the image build covers this. It does not, and it also is not affected by the pin. All 47 Dockerfiles install with:

COPY pyproject.toml uv.lock uv-requirements.txt ./
RUN ... uv sync --python 3.13 --frozen --no-install-project --no-dev --no-editable

uv sync --frozen resolves from uv.lock and treats pyproject.toml as advisory. Since the lock is the pin's own source of truth, images already receive the pinned versions and are unchanged by release-time pinning — verified by running the exact Dockerfile command against a pinned tree (exit 0, same package set). The pin is load-bearing only for PyPI, because uvx ignores the uv.lock inside an sdist — and 57 of 59 sdists do ship one, which makes the opposite easy to assume.

So a container-image check would not substitute for this; the wheel is the artifact that needs asserting.

Related

• fix: pin all dependencies in the pyproject.toml file #2856 — adds the pin-dependencies step this would exercise
• bug: four MCP servers ship test/lint tooling as runtime dependencies on PyPI #4458 — four packages shipping test tooling as runtime deps; the optional assertion above would prevent regressions once fixed