Skip to content

fix: allow installer to work on PEP 668 externally managed Python - #19

Merged
limingyaoo merged 1 commit into
mainfrom
fix-pep668-installer
Nov 11, 2026
Merged

limingyaoo merged 1 commit into
mainfrom
fix-pep668-installer

Conversation

@limingyaoo

Copy link
Copy Markdown
Member

Issue #, if available:

Fixed #13

Description of changes:

  • installer.py: run the three pip install calls with PIP_BREAK_SYSTEM_PACKAGES=1 set in the subprocess env (pip_env = dict(os.environ, PIP_BREAK_SYSTEM_PACKAGES="1")). Nothing else in the installer changes: same packages, same --user install location, and existing env vars such as ALL_PROXY are still passed through.
  • test/test_installer.py (new): unit tests for the no-proxy, HTTP proxy and HTTPS proxy paths. They check that pip gets the env var, that pip does not get the --break-system-packages flag, and that the proxy settings are kept.
  • .github/workflows/python.yml: new installer-on-os-python job. It runs installer.py on each distro's own Python (Debian 13/12, Ubuntu 24.04/22.04/20.04) as a non-root ggc_user through sudo ... sh -c, the same way Nucleus runs lifecycle scripts, and then checks that boto3 and awsiot can be imported.

Why is this change necessary:

With UseInstaller: true, the install step fails on distros that mark the system Python as externally managed (PEP 668), such as Debian 12+, Ubuntu 23.04+ and Raspberry Pi OS Bookworm+. pip refuses --user installs there with error: externally-managed-environment, so the component goes BROKEN.

#14 fixes this by adding --break-system-packages, but that flag only exists in pip 23.0.1 and later. Older pip exits with no such option: --break-system-packages. That would break installs that work today on Ubuntu 20.04/22.04, Debian 11 and AL2023.

pip reads every option from a PIP_<NAME> env var as well, and older pip skips env vars it doesn't recognize (if option is None: continue in cli/parser.py). So the env var gives the same override on pip 23.0.1+ and does nothing on older pip.

The existing CI can't catch either failure: it never runs installer.py, and actions/setup-python installs a Python that isn't externally managed. The new job closes that gap.

How was this change tested:

  1. pip version sweep (python:3.9 with an EXTERNALLY-MANAGED marker added): the env var is accepted on every pip from 19.3.1 to 25.3 and lifts the restriction on 23.0.1+. The flag is rejected on every pip from 19.3.1 to 23.0.

  2. Distro matrix. Each installer variant was run as ggc_user with sudo -n -E -H -u ggc_user -- sh -c, followed by an import check:

    Distro pip Current code fix: installation failures on PEP 668-compliant systems #14 This PR
    Debian 13 25.1.1 ❌ externally-managed ✅ ✅
    Debian 12 23.0.1 ❌ externally-managed ✅ ✅
    Ubuntu 24.04 24.0 ❌ externally-managed ✅ ✅
    Ubuntu 22.04 22.0.2 ✅ ❌ no such option ✅
    Ubuntu 20.04 20.0.2 ✅ ❌ no such option ✅
    Debian 11 20.3.4 ✅ ❌ no such option ✅
    AL2023 21.3.1 ✅ ❌ no such option ✅

    On every distro, packages were installed into ~ggc_user/.local and no files under the system Python directories were modified.

  3. Proxy. A tinyproxy HTTP proxy and an stunnel HTTPS proxy were set up on an isolated Docker network where the proxy was the only route to the internet. On Debian 13 and Ubuntu 22.04, this PR installed through the proxy on both the HTTP and the HTTPS (--cert bundle) path; the proxy log shows the PyPI connections. The current code failed on Debian 13 and fix: installation failures on PEP 668-compliant systems #14 failed on Ubuntu 22.04, matching the matrix above.

  4. End to end with a real Nucleus (2.18.4, offline, lifecycle copied from the published recipe, local stand-ins for TES and CloudWatch):

    Case Result
    Debian 13, current code ERRORED ×2, then BROKEN (externally-managed-environment)
    Ubuntu 22.04, fix: installation failures on PEP 668-compliant systems #14 ERRORED ×2, then BROKEN (no such option)
    Debian 13, this PR RUNNING; metric published over IPC reached PutMetricData; status reply success
    Ubuntu 22.04, this PR RUNNING; same full round trip
  5. CI: flake8 reports 0 errors and pytest passes 37 tests (34 existing plus 3 new). The new unit tests fail against the current code and against fix: installation failures on PEP 668-compliant systems #14. I also ran the steps of the new installer-on-os-python job locally on all five images: they pass with this PR, fail on Debian 13/12 and Ubuntu 24.04 with the current code, and fail on Ubuntu 22.04/20.04 with fix: installation failures on PEP 668-compliant systems #14. This PR is the first run of that job on GitHub Actions.

Any additional information or context required to review the change:

  • The change only affects UseInstaller: true. The recipe and the v4.0.0 default (UseInstaller: false) are unchanged.
  • --break-system-packages only lifts pip's PEP 668 check. Combined with --user, packages go to the component user's ~/.local, so files managed by apt/dpkg are never touched. This is the same install location the installer already uses on distros without PEP 668.
  • A venv under {work:path} (suggested in Installer.py: Cloudwatch breaks on Trixie Linux #13) would isolate dependencies better, but it needs a recipe change and the python3-venv package on Debian/Ubuntu. That's a possible follow-up, not a reason to hold this fix.
  • Unrelated, pre-existing problem: on AL2 (Python 3.7), UseInstaller: true already fails with or without this PR, because the newest awsiotsdk pip resolves requires awscrt==0.27.4, which has no Python 3.7 build.
  • aws-greengrass-device-defender has the same installer pattern; I'll open a separate PR there.

Checklist:

  • Updated the README if applicable
  • Updated or added new unit tests
  • Updated or added new integration tests
  • Updated or added new end-to-end tests
  • If your code makes a remote network call, it was tested with a proxy
  • You confirm that the change is backwards compatible

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Comment thread .github/workflows/python.yml Fixed
Comment thread test/test_installer.py Outdated
Comment on lines +13 to +21
def run_installer(self, monkeypatch):
calls = []
monkeypatch.setattr(subprocess, 'check_call', lambda cmd, **kwargs: calls.append((cmd, kwargs)))
try:
runpy.run_path(INSTALLER_PATH)
except SystemExit as e:
assert e.code == 0
assert len(calls) == 1
return calls[0]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question about test coverage on the proxy paths: only test_install_allows_externally_managed_python checks '--break-system-packages' not in cmd (and '--user' in cmd). The HTTP and HTTPS proxy tests check the env var and ALL_PROXY but not the flag.

As an experiment, I added '--break-system-packages' to just the two proxy-branch calls (lines 42 and 53) and kept env=pip_env. All three tests still passed. That installer would fail with no such option on pip < 23.0.1 behind a proxy, which is the regression this PR is meant to prevent.

Would you be open to moving the env-var, no-flag and --user assertions into run_installer, so all three paths check them? Or is there a reason to keep them only on the no-proxy test?

alter-mage
alter-mage previously approved these changes Nov 11, 2026
pip refuses to install into the system Python on Debian 12+ and
Ubuntu 23.04+, even with --user. Set PIP_BREAK_SYSTEM_PACKAGES=1 for
the pip calls instead of passing --break-system-packages, which
pip < 23.0.1 rejects.

Add unit tests and a CI job that runs installer.py on each distro's
own Python, and limit the workflow's GITHUB_TOKEN to read-only.
@limingyaoo
limingyaoo merged commit d0ef05c into main Nov 11, 2026
9 checks passed
@limingyaoo
limingyaoo deleted the fix-pep668-installer branch November 11, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Installer.py: Cloudwatch breaks on Trixie Linux

4 participants