Allow the use of arbitrary Pyodide versions #2002
Conversation
|
The Windows test failures are unrelated. I'll try to fix them later in the day, but happy to step back if someone else does it before me, or wishes to. |
Yes, exactly. |
If a package uses numpy or scipy at build time, it may be sensitive to the specific Pyodide version and not just the Python minor version. But only insofar as it depends on a specific numpy/scipy version, and this dependency should be clear from its |
That's right, if |
Yes, assuming that we first determine the |
|
Thanks for the responses @hoodmane and Pyodide folks! So I think the next thing to do would be to remove the implicit reliance on the host Python version, perhaps with python-build-standalone. That can be a follow-up PR, no need to add that here.
That's cool, I was speaking hypothetically, as in, "once the ABI is stable".
Just so I understand this- is that because pyodide bundles these libraries? And is this just a build-time concern or would that also limit the compatibility of the built wheels? |
Yes.
I don't think it should limit compatibility of the built wheels beyond what they already say in their |
|
Based on these recent discussions, here's what I understand and propose:
|
Yeah, that's my proposal. That'll fix the issue with the implicit reliance on the version of the host python.
Not sure I'm on board with the build identifiers above, but I'd agree that we shouldn't have to run cibuildwheel twice in normal setups. But I don't see when such a setup would be required. My understanding is that we'd create a new build identifier with each minor version of Python, because each ABI would be accompanied with a bump to the minor version of Python at the same time. See this conversion between myself and @hoodmane above:
As such, your example above would look something like: steps:
- uses: pypa/[email protected]
env:
CIBW_PLATFORM: pyodide
CIBW_BUILD: "cp312-pyodide_wasm32 cp313-pyodide_wasm32"
CIBW_TEST_REQUIRES_PYODIDE: "<...>" # and so on
...That would be functionally equivalent to your initial example because pyodide would not change ABI within a Python minor version. |
|
Unfortunately, the current failure is a known bug: pyodide/pyodide-build#143 |
|
Unfortunately, the python-build-standalone issue with symlinks appears to be cropping up again, this time inside the The issue appears to be related to astral-sh/python-build-standalone#380 - previously (#2328) fixed by resolving the symlink before calling the binary. I'm not sure that would be possible in this case though, as EDIT- As @agriyakhetarpal notes, pyodide/pyodide-build#143 is the best reference. |
|
If the approach that @hoodmane has to fix this in |
|
Okay, I'll actually make that PR if it's a blocker here. |
|
I had a play with your proposed workaround here, @hoodmane - it kinda worked, but I had to do a couple extra things
However, I now see that, although the |
|
Ugh this workaround keeps getting more complicated. Thanks for testing it out! We apparently should add more test coverage in pyodide-build. |
Yeah that is surprising. Steps to reproduce? |
|
Actually, there might be something else going on. Still investigating... |
|
Well, I'm pretty confused. But I'll share the minimal repros I'm working with, in case they're helpful @hoodmane. # docker run -it --rm ghcr.io/astral-sh/uv:debian
uv run --python=python3.13 --with pyodide-build python <<'EOF'
import subprocess
from pathlib import Path
import sys
import os
import textwrap
import shutil
venv_dir = Path("/tmp/pyodidevenv")
try:
shutil.rmtree(venv_dir)
except Exception as e:
pass
def run(args, env=None):
result = subprocess.run(args, env=env)
if result.returncode != 0:
sys.exit(f'{' '.join(args)} returned {result.returncode}')
run(['pyodide', 'venv', venv_dir])
env = os.environ.copy()
env["PATH"] = str(venv_dir / "bin") + os.pathsep + env["PATH"]
print("we're running this pip:")
run(['which', 'pip'], env=env)
run(['pip', 'install', 'pytest'], env=env)
run(['pytest'], env=env)
EOFThis fails on debian - I get the encoding error I've tried inserting the workaround, as shown here- # docker run -it --rm ghcr.io/astral-sh/uv:debian
uv run --python=python3.13 --with pyodide-build python <<'EOF'
import subprocess
from pathlib import Path
import sys
import os
import textwrap
import shutil
venv_dir = Path("/tmp/pyodidevenv")
try:
shutil.rmtree(venv_dir)
except Exception as e:
pass
def run(args, env=None):
result = subprocess.run(args, env=env)
if result.returncode != 0:
sys.exit(f'{' '.join(args)} returned {result.returncode}')
run(['pyodide', 'venv', venv_dir])
# --WORKAROUND--
base_python = Path(sys.executable)
(venv_dir / "bin" / "python-host-link").symlink_to(base_python)
(venv_dir / "bin" / "python-host").unlink()
(venv_dir / "bin" / "python-host").write_text(
textwrap.dedent(f"""
#!/bin/bash
export PYTHONHOME={base_python.parent.parent}
exec {venv_dir / "bin" / "python-host-link"} -s "$@"
""")
)
(venv_dir / "bin" / "python-host").chmod(0o755)
# make the python3.12-host bin point to the python-host script
(venv_dir / "bin" / "python3.13-host").unlink()
(venv_dir / "bin" / "python3.13-host").symlink_to(venv_dir / "bin" / "python-host")
# --END WORKAROUND--
env = os.environ.copy()
del env["VIRTUAL_ENV"]
env["PATH"] = str(venv_dir / "bin") + os.pathsep + env["PATH"]
print("we're running this pip:")
run(['which', 'pip'], env=env)
run(['pip', 'install', 'pytest'], env=env)
run(['pytest'], env=env)
EOFBut on linux, this fails with the following error: Which is weird, as we're in a virtualenv, but pip doesn't think so. |
|
Thanks! I'll try it out when I get home and see if I can figure out anything that helps. |
|
This works as expected for me: Maybe I should try with the docker image. |
Description
This PR updates the Pyodide build procedure (see #1456) that is enabled with
CIBW_PLATFORM: "pyodide"(or with the--platform pyodideCLI equivalent) post the changes in pyodide/pyodide#4882, wherepyodide/pyodide-buildwas unvendored from the main Pyodide repository to accommodate faster updates and fixes.This means that the Pyodide version and
pyodide-buildare not going to be in sync going forward, and that the Pyodide xbuildenv to install must be inferred by the versions available to install bypyodide-buildthrough a recently addedpyodide xbuildenv searchcommand, which prints out this table:Tap to expand table
Alternatively, one may use
pyodide xbuildenv search --allto return both compatible and non-compatible versions. This would, however, be better received as JSON (please see pyodide/pyodide-build#28).Additionally, in this PR, support has been added for installing arbitrary Pyodide versions, or, more specifically, arbitrary versions for "Pyodide cross-build environments (xbuildenvs)" – though, only the ones that are supported for a given
pyodide-buildversion. This has been implemented through an environment variableCIBW_PYODIDE_VERSIONand an associated configuration variable in the schema (through a table implemented via pyodide/pyodide-build#26).The rationale behind this is that WebAssembly/Pyodide builds are already experimental, and it would be useful to not tie the available Pyodide version to the
cibuildwheelversion – this would be helpful for downstream projects (statsmodels/statsmodels#9343, scikit-image/scikit-image#7525, scikit-learn/scikit-learn#29791, and so on) to allow testing against Pyodide's alpha releases and/or for the case of greater reproducibility against Pyodide's older releases.cc: @hoodmane and @ryanking13 for visibility
Suggested CHANGELOG entry
Since I didn't find a way to add an entry without the pre-commit hook removing previous entries, I've added a few lines here based on the current state of this PR. Please feel free to suggest changes or modify these lines directly.