install: Support "pkg @ https://…/pkg.whl" syntax

[chrysn]

The syntax "pkg @ https://…/pkg.whl" for a requirement to install (which comes from PEP-0508) is documented to be supported, but not actually implemented (instead, putting in just the URI is a micropip-proprietary supported feature, but it lacks support for features, see #205).

Open questions:

• Does it actually work? (I just tested parts on a local native Python build; let's see what CI says and maybe I can work off artifacts to actually test this)
• Should this kick off a deprecation process for the original syntax? (Probably not, as some documentation needs updating, but maybe a note in the code is due that if the one version causes trouble, go to the other).

Closes: #205
Replaces: #253

[chrysn]

My testing procedure:

• python3 -m build and upload the wheel to my trusted CORS enabled location
• Go to https://pyodide.org/en/stable/console.html
• At browser console, do: pyodide.loadPackage("https://downloads.endor.at/micropip-0.10.2.dev9+g19bdfe3b5-py3-none-any.whl")
• In REPL, do import micropip; await micropip.install("aiocoap[prettyprint] @ https://downloads.endor.at/aiocoap-0.4.14.post0-py3-none-any.whl")
• micropip.list() shows pygments and other pretty-printing related packages

During testing, I had to rewrite things a bit, and ended up simplifying. I don't think that the previous code could ever do anything else but go for add_sheel, add_requirement_inner or raise (in particular because a specifier can only be present when there is no URL; essentially, the old code's re-parsing of req in the presence of a URL was what lost the information and caused #205 in the first place), but that'll be for reviewers (and tests) to check.

All in all I think that the code after this PR is even simpler than what was there before (let alone after what #253 would cause), and per @ryanking13's suggestion in #253 (comment), this would then be the route to go.

[edit, added:] … and force-pushed because CI got stuck when I pushed too fast before.

[ryanking13]

Thanks. The Requirement class that packaging provides supports parsing the name @ url format. So I guess it would work, but we will need to carefully double check. I am glad that the change required was quite minimal.

[ryanking13]

I think we need to check the parsed url after converting the str to Requirement here. We don't support non .whl suffixes.

[chrysn]

Before this patch, attempting to load a non-wheel (trying with a tarball) resulted in a 2-deep parsing error. With this PR, this doesn't change for the await micropip.install("https://.../...tar.gz") case, but attempting to install package @ https://…/package….tar.gz gives a clearer "Invalid wheel filename (extension must be '.whl'): 'aiocoap-0.4.14.tar.gz'" error. (Previously, using the @ syntax also gave the old error).

I don't think we need further checking: Running with a non-wheel gives an error either way, the error message gives a way forward, and the less we explicitly check and just let the errors be clear, the easier any later additions can be. (I don't think we'll need support for .tar.gz ever because there are platform-independent wheels, but then again, I've used Python long enough to remember when it laid eggs, so maybe something else comes along).

Old error

Traceback (most recent call last):
  File "/lib/python3.13/site-packages/micropip/_vendored/packaging/src/packaging/_parser.py", line 62, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
  File "/lib/python3.13/site-packages/micropip/_vendored/packaging/src/packaging/_parser.py", line 80, in _parse_requirement
    url, specifier, marker = _parse_requirement_details(tokenizer)
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/lib/python3.13/site-packages/micropip/_vendored/packaging/src/packaging/_parser.py", line 124, in _parse_requirement_details
    marker = _parse_requirement_marker(
        tokenizer,
    ...<5 lines>...
        ),
    )
  File "/lib/python3.13/site-packages/micropip/_vendored/packaging/src/packaging/_parser.py", line 145, in _parse_requirement_marker
    tokenizer.raise_syntax_error(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        f"Expected end or semicolon (after {after})",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        span_start=span_start,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/lib/python3.13/site-packages/micropip/_vendored/packaging/src/packaging/_tokenizer.py", line 167, in raise_syntax_error
    raise ParserSyntaxError(
    ...<3 lines>...
    )
micropip._vendored.packaging.src.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after name and no valid version specifier)
    https://files.pythonhosted.org/packages/28/76/de52f7fa51ddbb5255f7b80d8965903bfa718420ac5d21b0852f5d81c1b1/aiocoap-0.4.14.tar.gz
         ^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.13/site-packages/micropip/package_manager.py", line 198, in install
    await transaction.gather_requirements(requirements)
  File "/lib/python3.13/site-packages/micropip/transaction.py", line 75, in gather_requirements
    await asyncio.gather(*requirement_promises)
  File "/lib/python3.13/site-packages/micropip/transaction.py", line 98, in add_requirement
    return await self.add_requirement_inner(Requirement(req))
                                            ~~~~~~~~~~~^^^^^
  File "/lib/python3.13/site-packages/micropip/_vendored/packaging/src/packaging/requirements.py", line 38, in __init__
    raise InvalidRequirement(str(e)) from e
micropip._vendored.packaging.src.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    https://files.pythonhosted.org/packages/28/76/de52f7fa51ddbb5255f7b80d8965903bfa718420ac5d21b0852f5d81c1b1/aiocoap-0.4.14.tar.gz
         ^

New error when installing with @ syntax

>>> await micropip.install("aiocoap @ https://files.pythonhosted.org/packages/28/76/de52f7fa51ddbb5255f7b80d8965903bfa718420ac5d21b0852f5
d81c1b1/aiocoap-0.4.14.tar.gz")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.13/site-packages/micropip/package_manager.py", line 205, in install
    await transaction.gather_requirements(requirements)
  File "/lib/python3.13/site-packages/micropip/transaction.py", line 74, in gather_requirements
    await asyncio.gather(*requirement_promises)
  File "/lib/python3.13/site-packages/micropip/transaction.py", line 91, in add_requirement
    return await self.add_requirement_inner(as_req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.13/site-packages/micropip/transaction.py", line 199, in add_requirement_inner
    wheel = WheelInfo.from_url(req.url)
  File "/lib/python3.13/site-packages/micropip/wheelinfo.py", line 92, in from_url
    name, version, build, tags = parse_wheel_filename(file_name)
                                 ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/lib/python3.13/site-packages/micropip/_utils.py", line 86, in parse_wheel_filename
    return parse_wheel_filename_orig(filename)
  File "/lib/python3.13/site-packages/micropip/_vendored/packaging/src/packaging/utils.py", line 98, in parse_wheel_filename
    raise InvalidWheelFilename(
        f"Invalid wheel filename (extension must be '.whl'): {filename!r}"
    )
micropip._vendored.packaging.src.packaging.utils.InvalidWheelFilename: Invalid wheel filename (extension must be '.whl'): 'aiocoap-0.4.14
.tar.gz'
>>> 

[ryanking13]

Ohh, okay. Thanks for the traceback. That makes sense.

[ryanking13]

Hey @chrysn. I just wanted to let you know that I did not forget this issue. Looking at the error, I think there are a few other places that need to be modified. I will take a look when I have time.