Plans for the initial version 0.1.0

[travishathaway]

Welcome to the conda-rlock project (issue #1!). This issue outlines what I would like to establish as the initial feature set for this conda plugin.

Here's a rough outline of what I'd like to include:

• Include a setting that users can toggle on and off for creating lock files (e.g. plugins.conda_rlock_enabled)
• Enable lock file creation/updating after the following commands are run:
  • create
  • update
  • install
  • remove
• Pixi should be able to recreate the environment using the lock file generated by this plugin
• PyPI dependencies should also be present in this lock file

Subsequent versions will add the ability to create environments using this lock file.

Good to know

The discussion below is primarily a conversation with myself, so I don't forget things I have learned or want to do next. Others can chime in too though.

[travishathaway]

This is the function I will want to make use of in rattler:

• https://github.com/conda/rattler/blob/a7370b5a304c32883e7beefed43f810305008434/crates/rattler_conda_types/src/prefix_record.rs#L257

That will help collect all the conda dependencies for an environment.

For the PyPI dependencies, I will need to use this function in rip:

• https://github.com/prefix-dev/rip/blob/6225bfbfc8d56ea130f0614f6a42d48a0ded1c78/crates/rattler_installs_packages/src/python_env/distribution_finder.rs#L64

[travishathaway]

pypi dependencies

Getting these is a little more difficult than the conda ones. To get them, I need the following information about the environment:

• Root location of it
• Python version it is currently using

To get the Python version, I just need to see whether Python is installed as a conda dependency and then get the version from that. If Python isn't even installed in the conda environment, than I don't have to worry about search for pypi dependencies 😉.

[travishathaway]

pypi dependencies continued...

So, I'm realizing this is going to be a lot more difficult to do than originally expected, and I am going to have to learn more about the intricacies of how PyPI packages are structured. The road block I ran into specifically was not being able to retrieve the following information about a PyPI package:

• location the URL where this is stored
• hash hash of the archive file the URL points to
• requires_dist requirements of the package (i.e. its dependencies)
• requires_python the python version this package requires
• editable if the project should be installed in editable mode (pip install -e I think)

It looks like there's a function in pixi that "knows how" to fetch all this information:

• https://github.com/prefix-dev/pixi/blob/3f261369387f0f1c0878d836cd6d04eff38adde8/src/lock_file/resolve/pypi.rs#L637

I don't want to depend directly on pixi for this, so I will instead attempt to copy it over to my project. This means I'll also need to copy over everything it depends on. Let's see goes. While doing this, I should avoid depending on the uv crates listed in pixi because this are not officially available via cargo. Instead, I should take a look around at other available crates for getting information about Python packages.

I also noticed there's a TODO at the bottom of the function for adding extras in the lock file. Maybe this is something I could implement and upstream later? We'll see...

[travishathaway]

Learning more about pypi dependencies

Since my last update, I have learned even more about pypi dependencies but also about some other things happening with rattler. I will attempt to summarize everything I've learned so far....

What's going on with rattler?

First, I've had some discussions with the rattler maintainers, and they have decided it would make sense to create a new rattler_pypi_interop crate. This crate would make it unnecessary to depend on rip (i.e. the rattler_installs_package because we would essentially just be copying over the functionality need from it. But, there are still some ongoing discussions around that here:

• Add a rattler_pypi_interop crate conda/rattler#1044

What have I learned about pypi packages?

This was a great read to learn about the limitations of rip and why prefix abandoned it in favor of uv:

• https://prefix.dev/blog/uv_in_pixi

Beyond those high level details, I've begun learning more about how to retrieve the metadata necessary for creating a lock file for pypi dependencies. Unlike with conda, all the information necessary simply isn't available on the filesystem. From what I've gathered, it seems necessary to make a request to retrieve something similar to repodata.json from pypi.org (more info here). Without this, I have no idea how I would retrieve information such as location, hash, etc. In pixi this is already done in the process of installing the pypi package, but for this tool, I will need to download this information myself.

This presents a couple of problems though:

• How do I know which pypi index to use?
  • This is isn't available on anywhere because it's usually provided at install time for tools like uv and pixi. Because I will need to obtain this information from the user, I will need to ask for this and default to https://pypi.org/simple. This means that I will not be able to handle the use case where pypi packages have been installed from multiple packaging indices.
• How much will this slow down the creation of lock files?
  • Not sure, but obviously introducing a step to download a package index of many MB in size will always slow things down

Next steps

Using what I've copied over to rattler_pypi_interop, create a simple program that is able to create that index cache and look up the packages necessary for the lock file entries I need. For usage examples, check out the original rip code.

Some other useful links:

• https://packaging.python.org/en/latest/specifications/binary-distribution-format/
• https://docs.pypi.org/api/index-api/

[tdejager]

Just as a note, PyPI does not have something like a repodata.json, as in the index in one file. It's split up and most clients use the /simple API to get this. This includes needing to get the entire wheel and reading the metadata from there or even needing to build from source but things like: https://peps.python.org/pep-0643/ have made this easier.

[tdejager]

Other relevant PEP's: https://peps.python.org/pep-0658/, https://peps.python.org/pep-0714/