Add support for building Android wheels #2349
Conversation
|
Let us know if you need (the rest of) CI triggered. :) |
|
@henryiii: This is close to being complete, so please enable the rest of CI. |
|
To run the integration tests on most of the CI machines, it looks like I'll need to automate installation of the correct Python version. For macOS this can be done the same way as iOS, but for Linux there's no existing code to reuse, because the native Linux build uses Docker. So I'll probably implement something that uses python-build-standalone, unless anyone has another suggestion. Apart from that, I think this PR is complete enough now that it's worth reviewing. @freakboy3742 and anyone else who's interested. |
|
python-build-standalone is fine, in fact, I'd like to use that for pyodide in the future, too. |
|
I've already written code to install a version of python-build-standalone in #2002, along with version pinning etc. I think it would work nicely here too. |
README.md
Outdated
| <sup>³ Requires a macOS runner; runs tests on the simulator for the runner's architecture.</sup> | ||
|
|
||
| <sup>³ Requires a macOS runner; runs tests on the simulator for the runner's architecture.</sup><br> | ||
| <sup>⁴ Building for Android requires the runner to be Linux x86_64, macOS ARM64 or macOS x86_64. Testing is supported on the same platforms, but also requires the runner to either be bare-metal, or support nested virtualization. CI platforms known to meet this requirement are: GitHub Actions Linux x86_64.</sup><br> |
There was a problem hiding this comment.
According to this it should also be possible to run the emulator for testing on "Azure Pipelines macOS (Microsoft-hosted agent)". But I haven't tried it yet.
Should there be a recommendation / example to link against the static one if possible rather than bundle ? |
| call( | ||
| "patchelf", | ||
| "--set-rpath", | ||
| f"${{ORIGIN}}/{libs_dir.relative_to(path.parent)}", |
There was a problem hiding this comment.
On this line I get the following error in a private pybind11 project:
Repairing wheel...
+ python -c 'import sysconfig; print(sysconfig.get_config_var("ANDROID_API_LEVEL"), end="")'
+ patchelf --set-soname libc++_shared-cd110349.so /tmp/cibw-run-536mrs3q/cp313-android_x86_64/repaired_wheel/unpacked/myprivatelib_pkg.libs/libc++_shared-cd110349.so
+ patchelf --replace-needed libc++_shared.so libc++_shared-cd110349.so /tmp/cibw-run-536mrs3q/cp313-android_x86_64/repaired_wheel/unpacked/myprivatelib/somelib.cpython-313-x86_64-linux-android.so
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/cibuildwheel/__main__.py", line 511, in <module>
main()
~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/cibuildwheel/__main__.py", line 63, in main
main_inner(global_options)
~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/cibuildwheel/__main__.py", line 207, in main_inner
build_in_directory(args)
~~~~~~~~~~~~~~~~~~^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/cibuildwheel/__main__.py", line 372, in build_in_directory
platform_module.build(options, tmp_path)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/cibuildwheel/platforms/android.py", line 127, in build
repaired_wheel = repair_wheel(
build_options, build_path, build_env, android_env, built_wheel
)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/cibuildwheel/platforms/android.py", line 421, in repair_wheel
repair_default(android_env, built_wheel, repaired_wheel_dir)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/cibuildwheel/platforms/android.py", line 496, in repair_default
f"${{ORIGIN}}/{libs_dir.relative_to(path.parent)}",
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/pathlib/_local.py", line 385, in relative_to
raise ValueError(f"{str(self)!r} is not in the subpath of {str(other)!r}")
ValueError: '/tmp/cibw-run-536mrs3q/cp313-android_x86_64/repaired_wheel/unpacked/myprivatelib_pkg.libs' is not in the subpath of '/tmp/cibw-run-536mrs3q/cp313-android_x86_64/repaired_wheel/unpacked/myprivatelib'
My final project structure should look like this:
myprivatelib_pkg:
-----------------
myprivatelib
├─ __init__.py
├─ version.py
└─ somelib.cpython-313-x86_64-linux-android.so
In my CMakeLists.txt I use:
install(TARGETS somelib DESTINATION myprivatelib)
If I change it to:
install(TARGETS somelib DESTINATION .)
the "Repairing wheel" step works, but my project structure is wrong and my unit-tests are failing.
That would be unsafe when a package has multiple .so files, for the reasons given here. In theory it may even be unsafe if multiple packages each link against their own separate libc++.so, and they pass C++ objects between them. This exact issue wouldn't arise on Linux, because the manylinux standard requires the C++ library to be provided by the OS. However, similar issues could affect other C++-based libraries if auditwheel includes multiple copies in multiple packages. I think the reason why we usually get away with this on Linux is that separate packages usually interact with each other via Python interfaces rather than C++ ones, so the crash scenarios described in the above link do not occur. The same should be true on Android. |
mhsmith
requested review from
hoodmane,
ryanking13 and
agriyakhetarpal
as code owners
This is very strange. It's evidently running Python, as we see it loading several standard library modules, but it doesn't show any Python stdout or stderr. Despite that, the test apparently passes, and then the logcat process returns non-zero. This non-zero return is normal when the emulator is shutting down, but since the testbed hasn't seen any output from Python, it treats this as a fatal error. The same thing has happened in CI on this PR, but only once. I've never seen it in the CI of Python itself, which uses the same testbed and runs many times every day. @joerick: Does it still fail with the current version of this PR, and does it fail the same way every time you try? |
|
Ah, now I think I see. Your test script is just I should probably change the testbed so it ignores a non-zero return from logcat if it's seen anything that looks like a valid logcat message, whether from Python or not. That still doesn't explain why this problem happened once in CI in a test which actually was producing output. Maybe the emulator shut down before the output could be read. The output should have occurred before the "TestRunner: finished" message, but it came from a separate thread, so maybe the message ordering isn't guaranteed. If this happens again, I'll try making the testbed pause for a moment after the test script completes before reporting the result. |
There was a problem hiding this comment.
Pull Request Overview
Adds comprehensive support for building and testing Android wheels in cibuildwheel.
- Introduces a new Android platform implementation (
platforms/android.py) with environment setup, build, repair, and test logic. - Extends core option handling, typing, packaging, and default schemas to recognize
android. - Adds end-to-end tests, updates CI workflows, examples, and documentation to cover Android builds.
Reviewed Changes
Copilot reviewed 35 out of 37 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| cibuildwheel/platforms/android.py | New Android build, repair, and test logic |
| cibuildwheel/options.py | Make build_frontend always present, default to build |
| cibuildwheel/typing.py | Include android in PlatformName literal |
| cibuildwheel/util/packaging.py | Support ignoring version number in Android wheel tags |
| docs/platforms.md | Add Android platform guide |
| docs/options.md | Add Android-specific environment & option variables |
| docs/ci-services.md | Update example description to include Android |
| examples/github-deploy.yml | Extend GitHub Actions example for Android |
| azure-pipelines.yml | Install JDK for Android SDK tools |
| .github/workflows/test.yml | Enable KVM for Android emulator in CI |
| unit_test/options_test.py | Adjust default frontend test expectations |
| unit_test/main_tests/main_options_test.py | Test parsing of Android config settings |
| unit_test/architecture_test.py | Add arch_synonym tests |
| test/utils.py | Generate deterministic, Android-aware expected wheels |
| test/test_android.py | End-to-end Android wheel build & test scenarios |
Comments suppressed due to low confidence (4)
examples/github-deploy.yml:34
- The example workflow doesn’t include steps to install or configure the Android SDK (
ANDROID_HOME); consider adding commands to download the command-line tools, install necessary SDK packages, accept licenses, and setANDROID_HOMEbefore building Android wheels.
- os: android-intel
cibuildwheel/platforms/android.py:1
- [nitpick] The Android platform implementation exceeds 600 lines in one file. Splitting environment setup, toolchain generation, and build/test logic into smaller modules would improve readability and maintainability.
import csv
cibuildwheel/util/packaging.py:161
- There are no unit tests verifying
find_compatible_wheelbehavior for Android wheel tags. Adding tests for Android identifier patterns (e.g.,android_21_arm64_v8a) would ensure this logic remains correct.
if platform.startswith(("manylinux", "musllinux", "macosx", "android", "ios")):
docs/platforms.md:317
- [nitpick] The note about requiring
test-sourcesfor iOS was removed; consider reinstating guidance to copy test files into the testbed app viatest-sourcesso users don’t miss this required configuration on iOS.
The iOS test environment can't support running shell scripts, so the [`test-command`](options.md#test-command) value must be specified as if it were a command line being passed to `python -m ...`.
|
@freakboy3742 I rerequeted a review from you since it's still "requesting changes". |
Acknowledged - I'm currently at EuroPython; I'll try to take a look, but I might not get a chance until I return to my office next week. |
The corresponding CPython PR, from which the Android Python releases in build-platforms.toml are being generated, is python/cpython#132870.