Build both Linux libc flavours per architecture on one agent

[Saurabh Singh (saurabh500)]

Description

The sandbox publish pipeline fanned out into seven concurrent build jobs, six of which wanted self-hosted RUST pool agents. The pools could not satisfy that burst — run 169925 spent roughly 46 minutes waiting for an agent after preflight finished, longer than any individual build step in the run.

The manylinux and musllinux jobs for a given architecture already wanted the same pool, the same demands and the same setup steps, and they call one build script parameterized by container image. This merges each same-architecture pair into a single job that runs both container builds.

Pool	Before	After
RUST-X64-WUS3 (self-hosted)	3	2
RUST-ARM64-WUS3 (self-hosted)	3	2
hosted macOS-14	1	1
Total concurrent build agents	7	5

Self-hosted demand drops by a third. Wheel output is unchanged: still seven wheels, now from five build jobs.

The check that would have gone blind

build-mock-wheel-linux-host.sh ended with find … -name '*.whl' | wc -l and failed only on zero. With two invocations in one job, the second would count the first invocation's wheels and pass even if it produced nothing — a check that cannot fail.

It now takes a required EXPECTED_PLATFORM_TAG and asserts a wheel matching that tag exists. Required rather than optional, so a future caller cannot silently opt out. Each flavour also writes to its own output subdirectory, so the two builds cannot see each other's wheels at all.

Deviation from the original plan

The plan called for two published artifacts per merged job, keeping the existing artifact names. previewRun rejected that: the governed Linux job template permits exactly one artifact per job, named from ob_artifactSuffix. Both flavours are now gathered into one artifact directory before publishing. collect-wheels.ps1 recurses for *.whl, so UploadPython is unaffected.

Trade-off

The two container builds run serially instead of side by side, offset by dropping one duplicated agent setup pass per merged pair. Against ~46 minutes of queue wait, that is worth it whenever the pools are contended.

A strategy: matrix was considered and rejected — it spawns one agent per leg, so it reduces nothing.

Related Issues

Fixes #392

Follow-up tracked separately in #393 (cross-compiling the Windows ARM64 wheel from the x64 agent, which would take peak to 4). Deliberately not attempted here: it needs a real run to confirm the RUST-W22-SQL25 image has VC.Tools.ARM64, and cross-compiling removes the accidental smoke test that building natively gives us.

Validation

What I actually exercised:

• The platform-tag assertion, against the real script with the docker wrapper stubbed. Six cases, all as expected:

  Case	Expect	Result
  healthy manylinux build	0	✅ 0
  healthy musllinux build	0	✅ 0
  musllinux expected, build produced nothing, manylinux wheel already present	non-zero	✅ 1
  build produced nothing, dir empty	non-zero	✅ 1
  second build succeeds alongside first flavour's wheel	0	✅ 0
  EXPECTED_PLATFORM_TAG unset	non-zero	✅ 1

  I broke the mechanism rather than just reverting the fix: on the third case the old count-based check passes (blind) while the new one fails. That is precisely the blind spot merging introduces. The healthy cases confirm it does not fire on good input.

• previewRun on definition 2293 against this branch — returns runs/-1 / buildId=-1, so no real run was created. The expanded YAML confirms all four Linux builds are present with correct tags, and UploadPython's dependsOn resolves to the five surviving job names with no stale entries. The first previewRun attempt is what caught the one-artifact-per-job constraint.

• Caller audit. Grepped the whole repo for build-mock-wheel-linux-host — exactly four call sites, all in the two merged jobs, all pass EXPECTED_PLATFORM_TAG. No caller missed by making it required.

What I did not exercise: the container builds themselves. A green previewRun proves the YAML expands and says nothing about whether the wheels build. This pipeline has publish side effects, so I did not queue a real run. First real run should be watched for the serial-build wall clock and for disk headroom, since a merged agent now holds two cargo target trees (gnu and musl land in different subdirectories, so they do not collide, but they do both occupy disk).

Checklist

• cargo bfmt passes — no Rust changed; pre-commit hook ran clean
• cargo bclippy passes — no Rust changed
• cargo btest passes — no Rust changed
• New/changed functionality has tests — shell assertion exercised in all six cases above, including the deliberately-broken one
• Public API changes are documented — none

---

Update: platform-tag assertion corrected after run 170165

The first real run of this branch (170165) failed in the merged Linux ARM64 job. The container
build succeeded and produced a wheel, then the new assertion rejected it:

ERROR: No mock wheel matching platform tag 'manylinux' in /mnt/vss/_work/1/a/wheels

Cause: scripts/build-mock-python-wheel-in-container.sh passes --auditwheel skip, so maturin
never retags glibc wheels as manylinux_*. They keep the default linux_<arch> tag. The
assertion expected a string that never appears, so it fired on healthy output. The x64 job had the
same defect and would have failed too; ARM64 just got there first.

This was my error, and it is worth naming how it got through: I validated the assertion against
wheel filenames I made up, rather than against real maturin output. The test double did not
reproduce real behaviour, so the passing case proved nothing.

Fixed by asserting on the tags actually emitted:

Job	Container	Expected tag
BuildLinux_x64	manylinux_2_34_x86_64	linux_x86_64
BuildLinux_x64	musllinux_1_2_x86_64	musllinux_1_2_x86_64
BuildLinux_arm64	manylinux_2_34_aarch64	linux_aarch64
BuildLinux_arm64	musllinux_1_2_aarch64	musllinux_1_2_aarch64

The two tags are mutually exclusive — musllinux_1_2_x86_64 does not contain linux_x86_64
(the linux in it is followed by _1_2_) — so a musl wheel cannot satisfy the glibc check. That
keeps the property the assertion exists for: the second invocation in a merged job still cannot
pass on the first invocation's wheels.

Verified against the real filenames recorded in runs 170138 and 170165, in both directions:

Case	Result
healthy manylinux vs linux_x86_64	exit 0
healthy musllinux vs musllinux_1_2_x86_64	exit 0
only manylinux present, musl expected	exit 1
only musllinux present, manylinux expected	exit 1
empty directory	exit 1

previewRun on definition 2293 re-validated against this branch: expands clean, UploadPython
still lists the five merged job names.

Note that previewRun cannot catch this class of bug at all — it proves YAML expansion and nothing
about what the containers produce. Only a real run confirms the fix.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 100%

🎯 Overall Coverage 92.6%

📦 Project: mssql-tds + mssql-odbc + mssql-py-core ℹ️ Note: diff coverage is reported, not enforced.

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

---

🔗 Quick Links

View Azure DevOps Build · Coverage Report