Python package manager

[JackReevesEyre-NOAA]

Seeking input on a software issue: how important is it to stick with a single python package manager across our shared codebases?

@pmav99 @tomsail @brey @SorooshMani-NOAA

uv was suggested as a good option for one project I'm working on, and I'm curious about:

1. Do you have experience with uv?
2. Does it matter if oceanmodeling repos have a mix of poetry, uv, and others?

Thanks for any insights!

[SorooshMani-NOAA]

I've heard about uv, but I've never used it. In general isn't it just the pkg manager? Does it affect project definition in any way? I understand that it affects the tests potentially when you use different package managers to install, but otherwise does it affect how a project is defined/built? The main nice thing about poetry is the lock file which uv seems to provide as well.

So in short I don't think there's any reason to switch unless there's a new feature in uv that your project really needs.

[JackReevesEyre-NOAA]

Thanks, Soroosh! Yes you're right - doesn't affect the package, but does affect the tests (and potentially special dependency sets for tests, dev, etc.).

No particularly strong reason to switch as far my project, I'm just curious for the reason of fitting in with other systems for hosting/serving.

[JackReevesEyre-NOAA]

Re-visiting this after some frustrations with poetry:

My understanding is that poetry doesn't install its own python version, so I've been trying to use conda to control the python version, then use poetry to build on top of this. A similar approach works great with pip, but seems to result in major headaches with poetry.

Is there an elegant way you manage python version(s) when developing your packages, @tomsail @pmav99 @SorooshMani-NOAA ?

If not, I'm becoming more tempted to try uv because it can handle the python version all in one. Plus it has easy conversion from pip....

[tomsail]

uv vs poetry

uv claims to be 100x faster than pip, that's why it becomes an interesting package manager when dealing with a big number of upstream dependencies.
I have never tried uv, because I haven't felt the need to build/install my package faster yet.

It now seems to support all the features that I like in poetry, like for example:

• relying on a lockfile
• build & publish

poetry still has a bigger community than uv (and more plugins maybe), but I don't think it really matters for the packages/repos we have in oceanmodeling.

When starting a new project, IMO it's up to the lead developer to choose the package manager. So if you want to go for uv, just put it clearly in the README that is it is the package manager used for that project.

And although it is possible to have multiple package managers for one project, best practice is to force other developers to install with the same package manager.

For downstream users, they'll just need to do pip install . either way, since it'll just use the pyproject.toml.

managing python versions and dependencies

I'm not sure your headaches will completely go away with uv if you have a problem with conda +pip/poetry.

@pmav99 has done an exhaustive guide a while ago that gives you a detailed procedure to install dependencies for a regular (non-root) user. Note that poetry is installed externally through pipx, and this is how you should install uv too.

Let us know if that could help you, otherwise I'll be happy to detail it more.

[JackReevesEyre-NOAA]

Thank you @tomsail -- I hadn't seen that guide before, but interestingly that's pretty much the work flow I had settled upon (conda base env -> conda env with required python version -> pip+venv with project dependencies).

However, when I try that same kind of thing with poetry instead of pip, it tries to install all the project dependencies inside the conda env (whichever one is activated, base or python version). That is apparently by design, and I can't seem to bypass the behavior.

I will try again with a fresh Monday morning brain and see if I can get things working. Then I might have some follow-up questions on exactly how you use poetry.... Thanks!

[pmav99]

uv is a newer project. The main hype point is speed, which in my opinion is not the most important feature. Since it was mentioned, especially on CI, the choice of a package manager is not really important because caching is both faster and more reliable (e.g. when PyPi is down)

WRT the installation of python, IMHV managing the environment should be done at higher level than that of the python package manager. More specifically it is important to make the distinction between two different problems/domains:

1. Dependency resolution
2. Managing the "system/environment"

There are a lot of tools that try to tackle these problems individually (e.g. pip-tools) and some that try to tackle both of them but AFAI know most (all?) of them fall short as soon as the complexity of your stack increases.

To make it more concrete, as far as geospatial software goes, installing python is just a small part of the equation. What about GEOS, GDAL, eccodes and any other non-python dependency you may have? What will you do about those? How will you manage them? Will you use a different tool for managing GDAL and uv to install Python? No, using a single tool for managing all the binary dependencies is simpler. And under this perspective whether uv/poetry/whatever can install python or not is actually irrelevant.

So what about conda, isn't that the answer to all of our problems? It would be if conda was equally dev-friendly as pip. Unfortunately, with conda you can't install code from a fork, you can't install code from a Pull Request, you can't do editable installs etc.

---

However, when I try that same kind of thing with poetry instead of pip, it tries to install all the project dependencies inside the conda env (whichever one is activated, base or python version). That is apparently by design, and I can't seem to bypass the behavior.

Again, I would suggest trying to tackle this issue at a different level. A really CLI-friendly solution is direnv or some similar project. With these tools the proper environment will always be active and then things will just work. And per directory environments can be useful in a bunch of other contexts, too

PS. Since we are in this discussion, the most interesting project of the lot is https://pixi.sh/latest/ which tries to combine PyPI and conda in a single tool