Remove pytest pin

[vyasr]

Description

Contributes to rapidsai/build-planning#105.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[vyasr]

The failure I see here looks like pytest-dev/pytest-cov#693 (cf. pytest-dev/pytest-xdist#1211).

[vyasr]

I resolved the above issue by ignoring one error. The other failure looks to be due to our testing of the streaming executor for cudf-polars using dask in some way that is conflicting with the latest versions of pytest and xdist.

[gforsyth]

Going to note down a few things here that should not block this PR, but we might want to track moving forward (I'll open issues once I'm sure I've got a handle on all of this):

1. Many of the test runs in ci/test_python_cudf.sh pass the --cov-config argument onward to run_cudf_pytests.sh. It is not clear to me if all of these coveragerc files even exist, and there's not an immediately apparent way to determine which file (if any) is being used to set coverage settings for a given subproject.

This is made trickier because coverage will use the first vaild config file that it finds, starting with .coveragerc, then setup.cfg, tox.ini, then pyproject.toml.

It would be good to standardize on one choice (probably pyproject.toml) except for those cases where we deliberately need to override settings for a particular test run.

[vyasr]

One of the remaining issues is that it seems like more recent versions of pytest are making it harder to guarantee that the current path is not added to sys.path. Even when I use --import-mode=importlib CI is failing for pylibcudf tests because some files from the source tree for pylibcudf are being imported instead of the versions in the install tree. --import-mode=append gets past this issue, but then fails for us because pylibcudf has multiple test modules with the same name in different subdirectories. I think moving the tests out of the pylibcudf package directory is the best way to avoid this problem altogether.

[gforsyth]

@vyasr -- I think I've got all the pytest errors squashed here. The pandas-test job is still failing, but looks more like legitimate test failures (although I don't know why)

[vyasr]

After some offline discussion I added a few extra tests to the list of expected failures for pandas. It seems likely that the pandas version we are currently using has some issues with the latest pytest version. Those issues are likely fixed in the latest pandas release, so we can revisit these when we tackle #19099.

[KyleFromNVIDIA]

Is the || exit 1 here intentional? I see set -e above, so it shouldn't be necessary.

[KyleFromNVIDIA]

Same question about || exit 1.

I see in this case set -euo pipefail is not being set above. We should set it.

[bdice]

Related to Kyle's question, does set -eou pipefail work the same as set -euo pipefail? I figured -o pipefail had to be together in the option set.

[gforsyth]

no, it doesn't -- I would expect that to error, but given that this is here, it might just ignore the pipefaile entirely 😬

[bdice]

Ceci n’est pas une pipefaile, as they say

[gforsyth]

I can push up a few changes to address the review comments (and fix the bad pipefail)

[bdice]

Going to merge this once CI passes again, because it moves a lot of code and is likely to have merge conflicts (like the one I just resolved).

[bdice]

/merge