Cache behaviour of multiple invocations of pytest.main() with import-mode=importlib

[septatrix]

The docs state the following about running pytest.main() more than once in the same process:

Calling pytest.main() will result in importing your tests and any modules that they import. Due to the caching mechanism of python’s import system, making subsequent calls to pytest.main() from the same process will not reflect changes to those files between the calls. For this reason, making multiple calls to pytest.main() from the same process (in order to re-run tests, for example) is not recommended.

Does this limitation still hold when using --import-mode=importlib? I would assume that the imports are never added to the global namespace and are thus garbage collected once the tests have run. Subsequent runs should then import and compile those modules anew

[The-Compiler]

As you can trivially confirm with a

import sys

def test_module():
    assert __name__ in sys.modules

that assumption is wrong.

[septatrix]

Yeah you are right. I could not think of such a simple counter example.
This makes the docs even more confusing...

With --import-mode=importlib things are less convoluted because pytest doesn’t need to change sys.path or sys.modules, making things much less surprising.

...as it obviously changes sys.modules

[The-Compiler]

Looks like that spot was missed with #7870. I opened #12985 to fix that.

[septatrix]

Shouldn't it be possible to delete the module from sys.modules after the tests have run?

[RonnyPfannschmidt]

While possible it's also unsafe

Any data of those modules that's referred somewhere else may use old code and or different class or function objects

In particular registries are prone