sphinx.testing is undocumented

[goerz]

https://github.com/sphinx-doc/sphinx-testing is marked deprecated and points to sphinx.testing as a replacement. However, the only documentation for this is https://www.sphinx-doc.org/en/master/devguide.html#unit-testing, which does not provide any details.

My first assumption was that maybe it's sufficient to replace from sphinx_testing import with_app with from sphinx.testing import with_app, but that does not seem to be the case:

ImportError: cannot import name 'with_app' from 'sphinx.testing'

I can sort of figure out the use of sphinx.testing by looking at the source code for Sphinx' own tests, but nonetheless there should be proper documentation. The (closed) issue #3458 already pointed out that documentation is still a todo item.

[tk0miya]

Absolutely. It must be a task for us!

[tk0miya]

As a quick note:

• Please create tests/conftest.py with following content:

import pytest

from sphinx.testing.path import path

pytest_plugins = 'sphinx.testing.fixtures'


@pytest.fixture(scope='session')
def rootdir():
    return path(__file__).parent.abspath() / 'roots'

• It provides absolute path of root directory of sphinx projects for testing as rootdir(). With above settings, they are placed under tests/roots.
• Please put sphinx projects for testing under tests/roots/test-*. By default, test-root is used for testing.
• Please write your test code.

import pytest

def test(app):
    # app is a Sphinx application object for default sphinx project (`tests/roots/test-root`).
    app.build()

@pytest.mark.sphinx(buildername='latex')
def test_latex(app):
     # latex builder is chosen here.
     app.build()

@pytest.mark.sphinx(testroot='case1')
def test_case1(app):
    # app is Sphinx application for case1 sphinx project (`tests/roots/test-case1`)
    app.build()

@pytest.mark.sphinx(confoverrides={'master_doc': 'content'})
def test_confoverrides(app):
    # a Sphinx application configured with given setting
    app.build()

I'll make a document in detail if I have time...

[jnikula]

Proper documentation for sphinx.testing would really be appreciated.

I'm banging my head against the wall migrating just this simple @with_app usage from sphinx-testing:

@with_app(confdir=testdir, create_new_srcdir=True, buildername='text')

Looking at the sphinx.testing source, I can't find support for creating new srcdir or even having separate confdir and srcdir. The above "quick note" seems to assume some rigid test structure/layout I'm not interested in (tests/roots/test-*), and I don't really know what's recommendation and what's requirement.

[jnikula]

Getting off-topic, but is it actually SphinxTestApp that conflates confdir and srcdir while Sphinx itself supports split conf and src? This is certainly a step back from sphinx-testing.

[jnikula]

Also, with @with_app I could trivially set up two independent apps, build them both in the same unit test, and compare the outputs. How would I do that with the make_app fixture that kind of assumes one app per unit test?

[danieleades]

Is it worth pointing out the irony of a documentation generator having such poor documentation?
;)

[tk0miya]

Yes, We must admit our documentation is poor. It's real of our project. So we need your help. How about read the code and contribute?

[tk0miya]

Looking at the sphinx.testing source, I can't find support for creating new srcdir or even having separate confdir and srcdir.

Indeed, it's not supported. If you'd like to do that, please post a PR.

The above "quick note" seems to assume some rigid test structure/layout I'm not interested in (tests/roots/test-*), and I don't really know what's recommendation and what's requirement.

sphinx.testing is helper functions and classes for pytest. So there is no "requirements". You don't need to use any fixtures and decorators. Of course, you don't need to create "test-root" directories too.

[jnikula]

Looking at the sphinx.testing source, I can't find support for creating new srcdir or even having separate confdir and srcdir.

Indeed, it's not supported. If you'd like to do that, please post a PR.

I'm afraid it just gets too involved for me. The Sphinx code base isn't exactly easy to approach. And I mean no disrespect, that's pretty much the case for any non-trivial project. But here it's amplified by the fact that there's no documentation on how sphinx.testing is supposed to work and be used in the first place! Hence the bug at hand.

The end result is that I'm stuck with sphinx-testing for as long as it works, and after that I'm more likely to hack together something in my own project to work around the limitations.

I've been relying on sphinx-testing for years, and I didn't expect its features to be ditched in the curb without a straightforward migration path: sphinx-doc/sphinx-testing#11

[jnikula]

On a related note, I did the bulk of the work to convert the Linux kernel to use Sphinx. One of the main criticisms towards Sphinx since then has been this kind of disregard of backwards compatibility. The kernel as a project does not want to impose latest tool version requirements to its developers or users, so we've had to work around the differences between Sphinx versions over time. Obviously it's your choice to develop the project as you see fit, but please take this as feedback from your users. It's not fun to have to keep catching up and adapting. Old stuff should keep working, and migrating to new stuff should be trivial.