Add tests to avoid future regressions

[cliffckerr]

Is your feature request related to a problem? Please describe.
I, like many others, hit #13352, and would prefer if this sort of thing didn't happen in the future -- it's very inconvenient when the CI pipeline starts failing due to a broken dependency.

Describe the solution you'd like
Please include in Sphinx's testing suite regression tests to ensure that Sphinx updates don't break popular third party extensions. Or, if there does need to be a breaking change, give the maintainers of the other packages sufficient warning to provide a fix before the new release.

Describe alternatives you've considered
Um ... mkdocs? 😆

[AA-Turner]

Hi Cliff,

Thank you for the feedback.

The problem in this specific case is that the extension had broken code, by not importing all issued modules, but worked due to side effects of Python's import system. Due to trying (hard!) to preserve backward compatibility in path handling whilst moving from string paths to pathlib, we needed to reduce how much was imported by default in sphinx.util. Again, we kept compatibility here by using module level __getattr__. We do try hard to keep compatibility, and we have a very generous policy that requires a deprecated feature is removed not before the second major release after the deprecation.

The wider problem is that Sphinx has hundreds of extensions and themes, if not over 1,000. We have no control over any third party projects. Testing even 50 such projects would massively increase our consumption of CI resource and slow down speed of development. We have (as with most projects) a very small team, but unlike many projects we have hundreds of such third-party plugins that can do nearly anything to the build process -- including overriding private methods, relying on undocumented features, and other creative use of the internals.

I strongly encourage most projects to test with Sphinx master from git, which would have found this error. Indeed, the explicit testing we requested from a handful of downstream projects for the pre-releases did discover this error.

However, I do apologise for the trouble that this has caused you, your team, and your organisation. We are always grateful for contributions, and I would be happy to review PRs improving our test coverage (I have noted before that I am unhappy with the state of our testing, we have far too many integration tests and too few unit tests, but the design of the application makes the latter a little challenging).

Thanks,
Adam

P.S.

mkdocs

I'm sure you're aware Sphinx has supported markdown documents for some time!

A

[jayaddison]

@cliffckerr are you working on this already? I have some ideas/suggestions for how you could get started if not; but I also agree with the preceding commentary that the ecosystem is large, and the CI resource overhead added by this could be significant - so it could prove to have some drawbacks.

[cliffckerr]

Thanks for your replies @AA-Turner and @jayaddison! As the maintainer of several OSS projects (including relatively low-level ones that other libraries depend on), I do very much appreciate the challenge of trying to move forward without introducing breaking changes. Every change breaks someone's workflow, but my guess is that Sphinx extensions are a long-tailed distribution, and just capturing the top 5-10 extensions could potentially be a good compromise. (For example: if a new NumPy release breaks some random postdoc's code, I wouldn't care or be surprised; but if it broke pandas and scikit-learn, I'd be pretty annoyed.)

While unfortunately I won't have time to work on Sphinx tests myself for the foreseeable future, I do have a couple suggestions that hopefully wouldn't be too much work and would be helpful:

• In the docs, you already list a few lists of third-party Sphinx extensions. You could pick some threshold (100 stars? 1m PyPI downloads?) and focus on only those. To further reduce the CI load, the tests wouldn't need to be run on every PR, just once (even manually) prior to a version release. (If it's helpful, you can take a look at what I implemented for downstream testing for one of my libraries; definitely could be made more elegant, but it works.)
• Another option (less work for you, more work for other developers) could be to adopt some of NumPy's practices, including deprecation warnings, regular release schedule, clear guidance on where to get the latest dev version, coordination with maintainers of key extensions, compatibility list, etc. I may be missing it, but I didn't see this information in the docs on building extensions.

Hope that's helpful and thanks for your (enormous) work maintaining this critical package!

[jayaddison]

I think Sphinx does cover most of the items from the second (alternative) option already; certainly deprecation warnings, prereleases, co-ordination and communication with key projects at release (and prerelease) time.

The pylint has a form of tests it calls primer tests: they can run the current ref against a small or extensive selection of Python projects, and the tests report back on number of warnings/errors/etc -- and IIRC performance differences -- encountered compared to the baseline.