Skip to content

Add the ability to declare safe tools in a cross-build environment. #2317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Apr 7, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ Options
| | [`CIBW_ENVIRONMENT_PASS_LINUX`](https://cibuildwheel.pypa.io/en/stable/options/#environment-pass) | Set environment variables on the host to pass-through to the container during the build. |
| | [`CIBW_BEFORE_ALL`](https://cibuildwheel.pypa.io/en/stable/options/#before-all) | Execute a shell command on the build system before any wheels are built. |
| | [`CIBW_BEFORE_BUILD`](https://cibuildwheel.pypa.io/en/stable/options/#before-build) | Execute a shell command preparing each wheel's build |
| | [`CIBW_XBUILD_TOOLS`](https://cibuildwheel.pypa.io/en/stable/options/#xbuild-tools) | Binaries on the path that should be included in an isolated cross-build environment. |
| | [`CIBW_REPAIR_WHEEL_COMMAND`](https://cibuildwheel.pypa.io/en/stable/options/#repair-wheel-command) | Execute a shell command to repair each built wheel |
| | [`CIBW_MANYLINUX_*_IMAGE`<br/>`CIBW_MUSLLINUX_*_IMAGE`](https://cibuildwheel.pypa.io/en/stable/options/#linux-image) | Specify alternative manylinux / musllinux Docker images |
| | [`CIBW_CONTAINER_ENGINE`](https://cibuildwheel.pypa.io/en/stable/options/#container-engine) | Specify which container engine to use when building Linux wheels |
Expand Down
6 changes: 5 additions & 1 deletion bin/generate_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,9 +181,12 @@
musllinux-x86_64-image:
type: string
description: Specify alternative manylinux / musllinux container images
repair-wheel-command:
xbuild-tools:
description: Binaries on the path that should be included in an isolated cross-build environment
type: string_array
repair-wheel-command:
description: Execute a shell command to repair each built wheel.
type: string_array
skip:
description: Choose the Python versions to skip.
type: string_array
Expand Down Expand Up @@ -273,6 +276,7 @@
properties:
before-all: {"$ref": "#/$defs/inherit"}
before-build: {"$ref": "#/$defs/inherit"}
xbuild-tools: {"$ref": "#/$defs/inherit"}
before-test: {"$ref": "#/$defs/inherit"}
config-settings: {"$ref": "#/$defs/inherit"}
container-engine: {"$ref": "#/$defs/inherit"}
Expand Down
14 changes: 14 additions & 0 deletions cibuildwheel/options.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ class BuildOptions:
environment: ParsedEnvironment
before_all: str
before_build: str | None
xbuild_tools: list[str] | None
repair_command: str
manylinux_images: dict[str, str] | None
musllinux_images: dict[str, str] | None
Expand Down Expand Up @@ -718,6 +719,18 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:

test_command = self.reader.get("test-command", option_format=ListFormat(sep=" && "))
before_test = self.reader.get("before-test", option_format=ListFormat(sep=" && "))
xbuild_tools: list[str] | None = shlex.split(
self.reader.get(
"xbuild-tools", option_format=ListFormat(sep=" ", quote=shlex.quote)
)
)
# ["\u0000"] is a sentinel value used as a default, because TOML
# doesn't have an explicit NULL value. If xbuild-tools is set to the
# sentinel, it indicates that the user hasn't defined xbuild-tools
# *at all* (not even an `xbuild-tools = []` definition).
if xbuild_tools == ["\u0000"]:
xbuild_tools = None

test_sources = shlex.split(
self.reader.get(
"test-sources", option_format=ListFormat(sep=" ", quote=shlex.quote)
Expand Down Expand Up @@ -835,6 +848,7 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
before_build=before_build,
before_all=before_all,
build_verbosity=build_verbosity,
xbuild_tools=xbuild_tools,
repair_command=repair_command,
environment=environment,
dependency_constraints=dependency_constraints,
Expand Down
60 changes: 53 additions & 7 deletions cibuildwheel/platforms/ios.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import shutil
import subprocess
import sys
import textwrap
from collections.abc import Sequence, Set
from dataclasses import dataclass
from pathlib import Path
Expand Down Expand Up @@ -151,6 +152,7 @@ def cross_virtualenv(
build_python: Path,
venv_path: Path,
dependency_constraint_flags: Sequence[PathOrStr],
xbuild_tools: Sequence[str] | None,
) -> dict[str, str]:
"""Create a cross-compilation virtual environment.

Expand Down Expand Up @@ -178,6 +180,8 @@ def cross_virtualenv(
created.
:param dependency_constraint_flags: Any flags that should be used when
constraining dependencies in the environment.
:param xbuild_tools: A list of executable names (without paths) that are
on the path, but must be preserved in the cross environment.
"""
# Create an initial macOS virtual environment
env = virtualenv(
Expand Down Expand Up @@ -210,14 +214,52 @@ def cross_virtualenv(
#
# To prevent problems, set the PATH to isolate the build environment from
# sources that could introduce incompatible binaries.
#
# However, there may be some tools on the path that are needed for the
# build. Find their location on the path, and link the underlying binaries
# (fully resolving symlinks) to a "safe" location that will *only* contain
# those tools. This avoids needing to add *all* of Homebrew to the path just
# to get access to (for example) cmake for build purposes. A value of None
# means the user hasn't provided a list of xbuild tools.
xbuild_tools_path = venv_path / "cibw_xbuild_tools"
xbuild_tools_path.mkdir()
if xbuild_tools is None:
log.warning(
textwrap.dedent(
"""
Your project configuration does not define any cross-build tools.

iOS builds use an isolated build environment; if your build process requires any
third-party tools (such as cmake, ninja, or rustc), you must explicitly declare
that those tools are required using xbuild-tools/CIBW_XBUILD_TOOLS. This will
likely manifest as a "somebuildtool: command not found" error.

If the build succeeds, you can silence this warning by setting adding
`xbuild-tools = []` to your pyproject.toml configuration, or exporting
CIBW_XBUILD_TOOLS as an empty string into your environment.
"""
)
)
else:
for tool in xbuild_tools:
tool_path = shutil.which(tool)
if tool_path is None:
msg = f"Could not find a {tool!r} executable on the path."
raise errors.FatalError(msg)

# Link the binary into the safe tools directory
original = Path(tool_path).resolve()
print(f"{tool!r} will be included in the cross-build environment (using {original})")
(xbuild_tools_path / tool).symlink_to(original)

env["PATH"] = os.pathsep.join(
[
# The target python's binary directory
str(target_python.parent),
# The cross-platform environments binary directory
# The cross-platform environment's binary directory
str(venv_path / "bin"),
# Cargo's binary directory (to allow for Rust compilation)
str(Path.home() / ".cargo" / "bin"),
# The directory of cross-build tools
str(xbuild_tools_path),
# The bare minimum Apple system paths.
"/usr/bin",
"/bin",
Expand All @@ -235,10 +277,12 @@ def cross_virtualenv(

def setup_python(
tmp: Path,
*,
python_configuration: PythonConfiguration,
dependency_constraint_flags: Sequence[PathOrStr],
environment: ParsedEnvironment,
build_frontend: BuildFrontendName,
xbuild_tools: Sequence[str] | None,
) -> tuple[Path, dict[str, str]]:
if build_frontend == "build[uv]":
msg = "uv doesn't support iOS"
Expand Down Expand Up @@ -291,6 +335,7 @@ def setup_python(
build_python=build_python,
venv_path=venv_path,
dependency_constraint_flags=dependency_constraint_flags,
xbuild_tools=xbuild_tools,
)
venv_bin_path = venv_path / "bin"
assert venv_bin_path.exists()
Expand Down Expand Up @@ -414,10 +459,11 @@ def build(options: Options, tmp_path: Path) -> None:

target_install_path, env = setup_python(
identifier_tmp_dir / "build",
config,
dependency_constraint_flags,
build_options.environment,
build_frontend.name,
python_configuration=config,
dependency_constraint_flags=dependency_constraint_flags,
environment=build_options.environment,
build_frontend=build_frontend.name,
xbuild_tools=build_options.xbuild_tools,
)
pip_version = get_pip_version(env)

Expand Down
21 changes: 21 additions & 0 deletions cibuildwheel/resources/cibuildwheel.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -397,6 +397,21 @@
"description": "Specify alternative manylinux / musllinux container images",
"title": "CIBW_MUSLLINUX_X86_64_IMAGE"
},
"xbuild-tools": {
"description": "Binaries on the path that should be included in an isolated cross-build environment",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
],
"title": "CIBW_XBUILD_TOOLS"
},
"repair-wheel-command": {
"description": "Execute a shell command to repair each built wheel.",
"oneOf": [
Expand Down Expand Up @@ -566,6 +581,9 @@
"environment-pass": {
"$ref": "#/$defs/inherit"
},
"xbuild-tools": {
"$ref": "#/$defs/inherit"
},
"repair-wheel-command": {
"$ref": "#/$defs/inherit"
},
Expand Down Expand Up @@ -991,6 +1009,9 @@
"repair-wheel-command": {
"$ref": "#/properties/repair-wheel-command"
},
"xbuild-tools": {
"$ref": "#/properties/xbuild-tools"
},
"test-command": {
"$ref": "#/properties/test-command"
},
Expand Down
2 changes: 2 additions & 0 deletions cibuildwheel/resources/defaults.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ build-verbosity = 0

before-all = ""
before-build = ""
# TOML doesn't support explicit NULLs; use ["\u0000"] as a sentinel value.
xbuild-tools = ["\u0000"]
repair-wheel-command = ""

test-command = ""
Expand Down
43 changes: 43 additions & 0 deletions docs/options.md
Original file line number Diff line number Diff line change
Expand Up @@ -1043,6 +1043,49 @@ Platform-specific environment variables are also available:<br/>
[PEP 517]: https://www.python.org/dev/peps/pep-0517/
[PEP 518]: https://www.python.org/dev/peps/pep-0517/

### `CIBW_XBUILD_TOOLS` {: #xbuild-tools}
> Binaries on the path that should be included in an isolated cross-build environment.

When building in a cross-platform environment, it is sometimes necessary to isolate the ``PATH`` so that binaries from the build machine don't accidentally get linked into the cross-platform binary. However, this isolation process will also hide tools that might be required to build your wheel.

If there are binaries present on the `PATH` when you invoke cibuildwheel, and those binaries are required to build your wheels, those binaries can be explicitly included in the isolated cross-build environment using `CIBW_XBUILD_TOOLS`. The binaries listed in this setting will be linked into an isolated location, and that isolated location will be put on the `PATH` of the isolated environment. You do not need to provide the full path to the binary - only the executable name that would be found by the shell.

If you declare a tool as a cross-build tool, and that tool cannot be found in the runtime environment, an error will be raised.

If you do not define `CIBW_XBUILD_TOOLS`, and you build for a platform that uses a cross-platform environment, a warning will be raised. If your project does not require any cross-build tools, you can set `CIBW_XBUILD_TOOLS` to an empty list to silence this warning.

*Any* tool used by the build process must be included in the `CIBW_XBUILD_TOOLS` list, not just tools that cibuildwheel will invoke directly. For example, if your build invokes `cmake`, and the `cmake` script invokes `magick` to perform some image transformations, both `cmake` and `magick` must be included in your safe tools list.

Platform-specific environment variables are also available on platforms that use cross-platform environment isolation:<br/>
`CIBW_XBUILD_TOOLS_IOS`

#### Examples

!!! tab examples "Environment variables"

```yaml
# Allow access to the cmake and rustc binaries in the isolated cross-build environment.
CIBW_XBUILD_TOOLS: cmake rustc
```

```yaml
# No cross-build tools are required
CIBW_XBUILD_TOOLS:
```

!!! tab examples "pyproject.toml"

```toml
[tool.cibuildwheel]
# Allow access to the cmake and rustc binaries in the isolated cross-build environment.
xbuild-tools = ["cmake", "rustc"]
```

```toml
[tool.cibuildwheel]
# No cross-build tools are required
xbuild-tools = []
```

### `CIBW_REPAIR_WHEEL_COMMAND` {: #repair-wheel-command}
> Execute a shell command to repair each built wheel
Expand Down
4 changes: 3 additions & 1 deletion docs/platforms/ios.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,9 @@ iOS builds support both the `pip` and `build` build frontends. In principle, sup

## Build environment

The environment used to run builds does not inherit the full user environment - in particular, `PATH` is deliberately re-written. This is because UNIX C tooling doesn't do a great job differentiating between "macOS ARM64" and "iOS ARM64" binaries. If (for example) Homebrew is on the path when compilation commands are invoked, it's easy for a macOS version of a library to be linked into the iOS binary, rendering it unusable on iOS. To prevent this, iOS builds always force `PATH` to a "known minimal" path, that includes only the bare system utilities, plus the current user's cargo folder (to facilitate Rust builds).
The environment used to run builds does not inherit the full user environment - in particular, `PATH` is deliberately re-written. This is because UNIX C tooling doesn't do a great job differentiating between "macOS ARM64" and "iOS ARM64" binaries. If (for example) Homebrew is on the path when compilation commands are invoked, it's easy for a macOS version of a library to be linked into the iOS binary, rendering it unusable on iOS. To prevent this, iOS builds always force `PATH` to a "known minimal" path, that includes only the bare system utilities, and the iOS compiler toolchain.

If your project requires additional tools to build (such as `cmake`, `ninja`, or `rustc`), those tools must be explicitly declared as cross-build tools using [`CIBW_XBUILD_TOOLS`](../../options#xbuild-tools). *Any* tool used by the build process must be included in the `CIBW_XBUILD_TOOLS` list, not just tools that cibuildwheel will invoke directly. For example, if your build script invokes `cmake`, and the `cmake` script invokes `magick` to perform some image transformations, both `cmake` and `magick` must be included in your cross-build tools list.

## Tests

Expand Down
Loading
Loading