json changes? size limits? pypi package json incomplete

[chaddupuis]

Our installation was working well until a couple months ago? when we stopped being able to download past a certain version of x package - mostly from more popular packages. I think I've traced the issue down to incomplete json being downloaded, possibly related to this ( https://discuss.python.org/t/backwards-incompatible-change-to-pypi-json-api/17154 )

Right now if I have in [requirements]
numpy = ==1.17.3
This works fine

But as of a few weeks/months ago?
numpy = ==1.26.0, for example, or anything above 1.19.5 errors out with "No files match requirement".

(this also happens with transformers and a handful of other packages that were working fine up until a certain point)

In examining the json from https://pypi.org/pypi/numpy/json for example, it seems to have 1.26.1. It's not invalid, so I don't think it's a string/request size limit.

I'm still trying to debug this, but I'm wondering if you have any input on this issue?

[chaddupuis]

I'm still looking at this so I could be going in the wrong direction, but so far it seems potentially related to this section in utils.py in the parse_wheel_filename_function

   if dashes == 5:   
        build_part = parts[2]
        build_match = _build_tag_regex.match(build_part)
        if build_match is None:
            raise InvalidWheelFilename(
                f"Invalid build number: {build_part} in '{filename}'"
            )
        build = cast(BuildTag, (int(build_match.group(1)), build_match.group(2)))

Dashes is only 4 so we are getting an empty build from the else (!==5) variable on wheels? - and then above that the dashes -2 line is removing the build sections so parts[2] is the filename and not the build, but this may be intended...

>>> filename.split("-", dashes)[2]
'cp310'
>>> filename.split("-", dashes -2 )[2]
'cp310-cp310-musllinux_1_1_x86_64.whl'

[chaddupuis]

With a clean install I found an actual bug, but have yet to track it down (I'll submit a PR if I can find it).

If your requirements has, for example, a configuration for python3.6, 3.7 and 3.10 and a particular package says it requires >= 3.7 morgan mirror will error out and fail to run. If the python 3.6 segment is removed from morgan.ini and the packages stay the same, it runs and works properly.

[ido50]

Hi. You are correct, a recent version of morgan changed the behavior such that packages are mirrored only if a version that satisfies all required Python versions of the different environments is found. This was introduced to fix an issue I considered more severe, with the plan of fixing it in a future version. I should have a fixed version by the end of the week, at worst I'll have to do multiple passes for each Python version, at best I can keep this in a single pass.