Add a rattler_pypi_interop crate

[travishathaway]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

We would like to add better support to the rattler library for working with packages from PyPI. Much of this functionality exists the rip project, but because this is no longer being actively developed, we would like to bring it over to the rattler project.

Why is this needed?

Another project that could provide this functionality in Rust is the uv project, but unfortunately the crates that provide it are not available on crates.io and we're not sure if they ever will be made available there.

To continue publishing the rattler crates on crates.io, we need to ensure that all of its dependencies are available there.

We also need this crate to fill in the gaps for pypi interoperability. For example, the rattler_lock crate is currently unable to create lock files with pypi packages using components exclusively available from rattler. This is a bad developer experience, and we would like to fix it.

What should happen?

We should first focus on bringing components from the rattler_installs_packages crate to a new create in rattler called rattler_pypi_interop.

Additional Context

See original discussion surrounding this issue here:

• [Discussion] Native PyPi/Pip package support #833

[iamthebot]

Curious to hear thoughts from @wolfv / @baszalmstra on why Pixi chose to go with uv interop vs. continuing to invest in rip.

This functionality is definitely key to have in rattler IMO since many conda environments inherently have to deal with PyPi packages and clean interop makes everyone's lives easier; however, as this is now an official "conda" project it's worth making a strategic decision about whether to implement PyPi interop support here or invest in cleaning up the uv interop story since that has a lot of momentum.

[jacopoabramo]

@iamthebot

There is a blog post about this decision.

[travishathaway]

Would be great to help the uv folks get some of their crates published. Not sure how much work is involved their though.

In the meantime, I think it makes sense to keep pursing this crate and just keep it as light-weight as possible. When I wrote this issue, I was envisioning a library with "read-only" capabilities for pypi packages. This would primarily be used to fetch metadata about packages installed via pip or uv.

As I've said elsewhere, my immediate use case for it is being able to generate rattle_lock lock files with out the need of uv as a dependency.

But yes, maybe we should a take a step back and ask ourselves whether this is going to require more effort than just helping the uv folks publish their existing crates which already handle this use case.

[travishathaway]

Also, to help coordinate conda's approach to PyPI interoperability, I've created this thread on our Zulip server:

• https://conda.zulipchat.com/#narrow/channel/457607-general/topic/PyPI.20interop

I'm hoping that projects like conda-pypi would be able to take advantage of whatever we do here. Maybe they also have an opinion on whether its best to depend on uv and put pressure on them to publish or whether it makes sense to maintain something separately ourselves.

[tdejager]

The last time I inquired, they said they were not willing to support a stable API and such for now, and focus more on the application itself. This might change in the future but AFAIK astral is not going to publish its crates in the short-term :)

[tdejager]

I do have the say, as one of the authors as rip and working a lot on the uv integration in pixi. That the uv crates handle everything a lot more competently than rip has and this has only become more the case as its being heavily developed. So if there is a way that we could integrate, we should prefer that. That said, I think we could still make a useful crate for rattler anyways even if we do not rely on their code directly.

[jacopoabramo]

@tdejager

from your comment on the previous issue:

uv does not publish on crates.io though so managing the dependency can be a bit of a hassle, and disallows publish to crates if you are making use of the git dependency.

by bringing this new crate from rip, wouldn't there be a risk of re-creating the same problem previously encountered, as in "using rip instead of uv, which is less performant"?

I'm just curious of the situation be mindful, I'm in no way an expert on any of these projects so it's just food for thought

[tdejager]

@jacopoabramo I agree. It would make sense to keep this as lean as possible, just reading the dist-info folders, which should not be as performance depending like solving or querying the index a lot.

[travishathaway]

Linking this discussion because I think it's somewhat related to what we've been discussing here:

• Enable installation via cargo astral-sh/ruff#43