question: best practices for tests in independent backends

[kvemani]

What are the best practices for tests in backends that live outside the ibis repository? From our (@amueller and myself) exploration, currently, our understanding is that the set of backends is being inferred using the file structure within the repository (relevant code link here).

@datapythonista Please suggest. If any related changes can be a useful contribution to the Ibis repository, I will be happy to work on it. Thanks!

Edit 1: To add more context, I would like to run tests on the SQL Server backend, hosted here.

[datapythonista]

Let me try to summarize what I've got in mind (surely open to discussion, and in practice things can be more complex).

We've got 3 different kinds of tests I'd say:

• Core Ibis tests, in ibis/tests
• Generic backend tests, in ibis/backends/tests
• Tests specific to a backend, in ibis/backends/sqlite/tests for example (you'll have the mssql ones in your own repo)

If we agree on this, I don't think you care about the core ibis tests, and you are already running the tests specific to your backend. So, what is missing is the generic backend tests in ibis/backends/tests. Those are the kind of tests that should make sure that an expression like con.table('my_table')['my_column'] * ibis.literal(2) returns the expected results in all backends.

These generic backend tests were designed assuming all backends would be in this repo, and are not so generic in practice. They've got lots of decorators like @pytest.mark.only_on_backends(['postgres', 'mysql']). Before it was even worse, and instead of providing strings for the backends to include/exclude a class was provided, making required to have dependencies, running backend specific code...

Also, each backend has a TestConf class, with information about the backend, like how is the rounding done in the backend. This was moved to the backend directory, but that part is still a bit too coupled in my opinion.

To me, what would be ideal is:

• The generic backend tests are really generic. They literally don't know anything and don't make any assumption on which backends will be tested with those
• They can be released (with Ibis, or separately), and then installed from third-party backends
• For tests that should run conditionally, instead of specifying which backends should run the tests, the backends should specify which features/traits they implement, and the tests should run based on those
• TestConf should be removed, and these features/traits should be specified directly in the backends, and not in a test class for the backend. For example, knowing how rounding is performed for a backend is something that we should be able to ask the backend directly (by backend I mean the current Backend class, which is likely to be merged with the Client class of the backend).

All this is a decent amount of work. It'd be great if you can work on it, but it's difficult to know where to start. You've got #2564 you can work on, but there are not many other tickets well defined. I think checking which tests are run conditional depending on the backend, and trying to specify backend traits that could be used to know when to run the tests would be very helpful. And even better writing in an issue an "inventory" of conditional tests, which backends are included/excluded, and brainstorm on the different kinds of backends.

CC @tswast we'll eventually need this for bigquery too (and for OmniSci and others)

[amueller]

Thanks for a great summary, and I totally agree on the ideal outcome. I think to unblock @kvemani, we don't necessarily need these two yet:

For tests that should run conditionally, instead of specifying which backends should run the tests, the backends should specify which features/traits they implement, and the tests should run based on those

TestConf should be removed, and these features/traits should be specified directly in the backends, and not in a test class for the backend. For example, knowing how rounding is performed for a backend is something that we should be able to ask the backend directly (by backend I mean the current Backend class, which is likely to be merged with the Client class of the backend).

Which are probably the hardest for someone not intimately familiar with the project to approach.

They can be released (with Ibis, or separately), and then installed from third-party backends

I think releasing with Ibis would be fine, and having an interface to that would be the most important part to get us started.
This reminds me of the multi-year journey we had with sklearn for testing third party models.
You can see some of the outcome here:

https://scikit-learn.org/stable/modules/generated/sklearn.utils.estimator_checks.parametrize_with_checks.html#sklearn.utils.estimator_checks.parametrize_with_checks

This decorator yields all the tests that are applicable to a given model in sklearn via the tags.

For a first step, it's already useful to know what fails, and so having some generator or other mechanism that yields the tests so they can be run within the backend would be awesome.
Having a backend-specific configuration would be sugar on top for us.

[datapythonista]

Agree. Also happy to move forward with other solutions that are not the ideal, but fix our immediate problems, and don't make things more complex.

For example, I'm happy to work on the releasing of the tests, try them in master for mssql (and any other tihrd-party backend, and add stuff like @pytest.mark.only_on_backends(['bigquery', 'omnisci', 'mssql']) to the tests to make them pass in the third-party repo. This is clearly not the ideal final approach, but the time to market to get third-party backends better tested should be very fast, and the work on making Ibis backend-agnostic can be done later.

[amueller]

Awesome, I think that would be super helpful!

[cpcloud]

Factoring out the backend tests into a separate library might be worth the effort if it enables folks to validate third party backends easily. This would also trim the core of the ibis test suite considerably, allow it to focus solely on expressions.

The main downside here is that testing changes to ibis becomes more cumbersome.

I don't think we'd be ready to build this kind of library until we have more stability in the internals. We're planning on gutting some of the internals in 2022, so splitting out would have to happen after that.

[cpcloud]

xref: #2307

[cpcloud]

I think at this time we're not considering making the ibis backend test suite workable with arbitrary third party backends.