Repository navigation
fix: allow installer to work on PEP 668 externally managed Python - #19
Conversation
e598e4e to
e9ba84f
Compare
| def run_installer(self, monkeypatch): | ||
| calls = [] | ||
| monkeypatch.setattr(subprocess, 'check_call', lambda cmd, **kwargs: calls.append((cmd, kwargs))) | ||
| try: | ||
| runpy.run_path(INSTALLER_PATH) | ||
| except SystemExit as e: | ||
| assert e.code == 0 | ||
| assert len(calls) == 1 | ||
| return calls[0] |
There was a problem hiding this comment.
Question about test coverage on the proxy paths: only test_install_allows_externally_managed_python checks '--break-system-packages' not in cmd (and '--user' in cmd). The HTTP and HTTPS proxy tests check the env var and ALL_PROXY but not the flag.
As an experiment, I added '--break-system-packages' to just the two proxy-branch calls (lines 42 and 53) and kept env=pip_env. All three tests still passed. That installer would fail with no such option on pip < 23.0.1 behind a proxy, which is the regression this PR is meant to prevent.
Would you be open to moving the env-var, no-flag and --user assertions into run_installer, so all three paths check them? Or is there a reason to keep them only on the no-proxy test?
pip refuses to install into the system Python on Debian 12+ and Ubuntu 23.04+, even with --user. Set PIP_BREAK_SYSTEM_PACKAGES=1 for the pip calls instead of passing --break-system-packages, which pip < 23.0.1 rejects. Add unit tests and a CI job that runs installer.py on each distro's own Python, and limit the workflow's GITHUB_TOKEN to read-only.
e9ba84f to
61ebae6
Compare
Issue #, if available:
Fixed #13
Description of changes:
installer.py: run the threepip installcalls withPIP_BREAK_SYSTEM_PACKAGES=1set in the subprocess env (pip_env = dict(os.environ, PIP_BREAK_SYSTEM_PACKAGES="1")). Nothing else in the installer changes: same packages, same--userinstall location, and existing env vars such asALL_PROXYare still passed through.test/test_installer.py(new): unit tests for the no-proxy, HTTP proxy and HTTPS proxy paths. They check that pip gets the env var, that pip does not get the--break-system-packagesflag, and that the proxy settings are kept..github/workflows/python.yml: newinstaller-on-os-pythonjob. It runsinstaller.pyon each distro's own Python (Debian 13/12, Ubuntu 24.04/22.04/20.04) as a non-rootggc_userthroughsudo ... sh -c, the same way Nucleus runs lifecycle scripts, and then checks thatboto3andawsiotcan be imported.Why is this change necessary:
With
UseInstaller: true, the install step fails on distros that mark the system Python as externally managed (PEP 668), such as Debian 12+, Ubuntu 23.04+ and Raspberry Pi OS Bookworm+. pip refuses--userinstalls there witherror: externally-managed-environment, so the component goes BROKEN.#14 fixes this by adding
--break-system-packages, but that flag only exists in pip 23.0.1 and later. Older pip exits withno such option: --break-system-packages. That would break installs that work today on Ubuntu 20.04/22.04, Debian 11 and AL2023.pip reads every option from a
PIP_<NAME>env var as well, and older pip skips env vars it doesn't recognize (if option is None: continueincli/parser.py). So the env var gives the same override on pip 23.0.1+ and does nothing on older pip.The existing CI can't catch either failure: it never runs
installer.py, andactions/setup-pythoninstalls a Python that isn't externally managed. The new job closes that gap.How was this change tested:
pip version sweep (python:3.9 with an
EXTERNALLY-MANAGEDmarker added): the env var is accepted on every pip from 19.3.1 to 25.3 and lifts the restriction on 23.0.1+. The flag is rejected on every pip from 19.3.1 to 23.0.Distro matrix. Each installer variant was run as
ggc_userwithsudo -n -E -H -u ggc_user -- sh -c, followed by an import check:On every distro, packages were installed into
~ggc_user/.localand no files under the system Python directories were modified.Proxy. A tinyproxy HTTP proxy and an stunnel HTTPS proxy were set up on an isolated Docker network where the proxy was the only route to the internet. On Debian 13 and Ubuntu 22.04, this PR installed through the proxy on both the HTTP and the HTTPS (
--certbundle) path; the proxy log shows the PyPI connections. The current code failed on Debian 13 and fix: installation failures on PEP 668-compliant systems #14 failed on Ubuntu 22.04, matching the matrix above.End to end with a real Nucleus (2.18.4, offline, lifecycle copied from the published recipe, local stand-ins for TES and CloudWatch):
externally-managed-environment)no such option)PutMetricData; status replysuccessCI:
flake8reports 0 errors andpytestpasses 37 tests (34 existing plus 3 new). The new unit tests fail against the current code and against fix: installation failures on PEP 668-compliant systems #14. I also ran the steps of the newinstaller-on-os-pythonjob locally on all five images: they pass with this PR, fail on Debian 13/12 and Ubuntu 24.04 with the current code, and fail on Ubuntu 22.04/20.04 with fix: installation failures on PEP 668-compliant systems #14. This PR is the first run of that job on GitHub Actions.Any additional information or context required to review the change:
UseInstaller: true. The recipe and the v4.0.0 default (UseInstaller: false) are unchanged.--break-system-packagesonly lifts pip's PEP 668 check. Combined with--user, packages go to the component user's~/.local, so files managed by apt/dpkg are never touched. This is the same install location the installer already uses on distros without PEP 668.{work:path}(suggested in Installer.py: Cloudwatch breaks on Trixie Linux #13) would isolate dependencies better, but it needs a recipe change and thepython3-venvpackage on Debian/Ubuntu. That's a possible follow-up, not a reason to hold this fix.UseInstaller: truealready fails with or without this PR, because the newestawsiotsdkpip resolves requiresawscrt==0.27.4, which has no Python 3.7 build.aws-greengrass-device-defenderhas the same installer pattern; I'll open a separate PR there.Checklist:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.