Refactoring of testing infrastructure

[ByteB4rb1e]

The testing infrastructure for pipenv is serving its purpose well, but as the contributor base diversifies across different platforms and environments, certain limitations are surfacing. This proposal aims to refator the test suite to improve cross-platform compatibility, reduce maintenance overhead, and align with pypa conventions. By adopting tools like tox, modularizing workflows, and ensuring local executability, it tries to address critical pain points while enhancing accessibility for contributors, including those using unconventional environments like MSYS2.
My proposal is designed to uphold pipenv's commitment to a seamless developer experience while minimizing disruption for existing contributors.

The initiative for that came through PR #6373, where I struggled to set up the testing infrastructure within my development environment, which is MSYS2, an environment consisting of a POSIX middleware and a collection of common UNIX and GNU tools running natively under Windows.

I am a TUI user and am currently using MSYS2 as my daily driver, moving away from macOS at work and Ubuntu in private. The reason for that is because the majority of my colleagues and customers use Windows and I'd like to be able to support them better. Also, Windows in conjunction with MSYS2 has a few very unique value propositions. It enables me to stay within a POSIX environment, while using UNIX tooling on Windows natively and having access to Windows facilities such as the Windows Forms API. This allows me to rapidly integrate TUI applications for GUI users and makes me a better, less opinionated collaborator.

The first approach is to move (light) testing facilities like Linting and Vendoring away from pre-commit to tox (pypa) and just wrap it by pre-commit for CI and contributors that are unfamiliar with the tox interface. This allows for contributors like me to use their native development environments without virtualization. This is necessary due to the fact, that the author of pre-commit is unwilling to support environments such as MSYS2 out of personal preference, since they consider Windows to be the "wrong platform" and POSIX-emulation on Windows to be "the wrong convention". If there are any concerns for supporting the MSYS2, or other unconventional environments, I'd be happy to address them directly.

nox would be an alternative to tox, but I suggest tox over nox, as it uses static configuration (compared to nox using dynamic configuration) which hinders the introduction of side-effects.

In addition to migrating light testing facilities away from pre-commit, I suggest adapting the current CI (heavy) testing workflow to also be executable locally, without introducing additional maintenance overhead. This could be achieved through an integration with a python executable management tools like pythonbrew and pythonz, or pyenv, so that all tests (except for the matrix, multi-platform facilities) can be executed locally and therefore reduce the overall build minutes of the CI, in addition to giving contributors a bigger toolset for validating their code integrations.

I haven't yet grasped the development philosophy of pipenv, so I'm not able to make a recommndation yet on which of the previously mentioned tools for Python executable management aligns best with the pipenv methodology. Since tox allows for great modularization, the introduction of this sort of tooling is optional though, so wouldn't introduce any complexities for contributors unfamiliar with testing setups for multiple Python executable versions.

All of this would keep the testing environment contained within the Python and pypa ecosystem, in addition to removing the dependency of GitHub workflows (except for matrix multi-platform tests) in regards to testing.

Outline of steps I am proposing and will refine if the overall proposal is of interest to the project:

• resolve all open issues in regards to testing infrastructure
• migrate (light) testing orchestration from pre-commit to tox
• wrap tox by pre-commit
• migrate (heavy) testing from CI to tox, without breaking multi-platform testing
• wrap tox by CI

Kind regards

[matteius]

Hey @ByteB4rb1e thank you for taking the time to write this up and for helping with some things in the project recently. I am not sure I understand the motivation for adopting tox -- we are using pytest. pre-commit is the linting hooks, not the testing framework. Have you checked how the github CI sets up the pypiserver which hosts the local artifacts for testing non pypi installs? Id suggesting giving a read through: https://github.com/pypa/pipenv/blob/main/.github/workflows/ci.yaml