Interested in contributing a Sphinx community survey

[isabela-pf]

Hello, all! I’m reaching out on behalf of Quansight Labs members who work on documentation projects across the ecosystem. We use Sphinx on many of the projects we work on and would like to make time to contribute to the Sphinx community. We’re also interested in getting a better sense of who uses Sphinx (and who doesn’t) and what does and does not work well for them, so we’ve drafted the beginnings of a community survey. The data from this survey would be shared publicly and, ideally, used as a collective resource to make decisions about Sphinx and related documentation tooling easier by basing them on community feedback. I’ve been told that this previously came up in a meetup in 2024 (#12443).

Right now the survey it is set up to work with Typeform and has been written as a YAML file. We want this to be a collaborative effort and are looking for feedback to turn this draft into something the whole community wants to see.

If you want some examples of what publicly shared data looks like for projects like this, you can explore the jupyter/surveys repo (some of which I’ve worked on before).

Do you have a review process you prefer for non-code contributions like this? I am also happy to open a PR or schedule time to meet if that’s easiest.

I’ll check in again in a week if I don’t hear anything by then. Thank you in advance!

(I've also been told to @melissawm, @Carreau, @trallard, @humitos, @ericholscher, and @hugovk)

[ericholscher]

I don't have a ton of feedback other than maybe having some thoughts on the questions. Happy to give them a peek once we have them put together. Otherwise this seems great, thanks for the initiative!

Probably seems worth pinging @AA-Turner and other Sphinx folks as well?

[Carreau]

The current state of what we wrote is here (EDIT: copy here for now), we think that defining survey in a (yaml) document, seem to be easier for long term re-use and iteration/review, as it is quite annoying to have to test the survey.

We hope that with a yaml definition this will help iterate of fix via problems via PRs,
and potentially rerun the survey regularly without having to create full new form and reenter the questions by hand.

[hugovk]

... here ...

This is 404 for me, is this a private repo?

[Carreau]

This is 404 for me, is this a private repo?

Duh. Yes. I don't think I can make it non public.

https://gist.github.com/Carreau/ca0867056c55f1ea808516b929174d39 is a copy in the meantime.

[trallard]

Want me to make it public @Carreau ? I should be able to

[Carreau]

I actually had the right, I just did not found the button, It should be public now.

[chrisjsewell]

This is great cheers guys, yeh unfortunately I got pretty busy after #12443 and was able to push it forward 😅

But with you guys helping to do this I'm sure we can gather some really useful feedback 🙏

Haven't looked at your technical solution just yet, may have comments on that a little later

[AA-Turner]

Quansight Labs ... use Sphinx on many of the projects we work on and would like to make time to contribute to the Sphinx community.

This is very generous, thank you.

---

We’re also interested in getting a better sense of who uses Sphinx (and who doesn’t) and what does and does not work well for them. The data from this survey would be shared publicly and, ideally, used as a collective resource to make decisions about Sphinx and related documentation tooling easier by basing them on community feedback.

I have no opposition in principle to a survey, though I don't personally have the time to organise the administration, beyond approving the questions.

My initial reaction is that the issue tracker is a reasonable sample of what people currently care about, namely that (1) Sphinx is not fast enough, (2) autodoc and autosummary can be improved, (3) conf.py is too Python-specific / Sphinx is too Python-specific in general, and maybe (4) some edge-cases around toctree. Having a survey won't help us fix these, and we already know that they're problems.

Hence, if we ran a survey I would want to get different (actionable!) insights, especially from those that don't use Sphinx or have stopped using it. For example:

• I think Sphinx has very little exposure to the Open API ecosystem, or JS, or Rust. What do these projects use, and what can Sphinx do to effectively compete? Conversely, I think we have reasonable exposure to C/C++ -- but is this actually true, or just outdated intuition?
• Likewise, how much of an impact does the default markup language have? Should we make more effort to advertise that reStructuredText ≠ Sphinx? Is AsciiDoc worth thinking about?
• How much energy should we spend on supporting non-HTML output formats? My understanding is that MkDocs, docsify, docusarus, mdbook, rustdoc, etc only support HTML output. Should we adopt new output formats such as Typst?
• How valuable is "batteries included"? Several other tools have flashy websites and lots of bundled, but Sphinx has a very rich ecosystem of themes and plugins, and each documentation site can add site-specific customisation, which can produce (subjectively) nicer output.
• Is our "marketing" information lacking? Our website has been improved recently, but it is far less fancy than some competitors (especially those backed by firms created to support the tool).
• Who are our competitors? I certainly think we compete in the general "documentation generator" category, but Sphinx is at heart a static-site generator, just one with lots of built-in cross-referencing support. Should more effort be invested into extensions such as ablog to better compete with Hugo, Gatsby, Jekyll?
• One USP of Sphinx is intersphinx, but what are the others? How can we better capitalise on these?

As much as this information is nice to know, though, it would be good to have agreed goals for what information we're seeking, and what decisions this will inform. I think such a survey should also make efforts to target both open-source (where we have a lot of visibility) and use in industry (where I'm aware Sphinx and competitors are used, but we have far less insight). Survey design is not my speciality, though!

---

Do you have a review process you prefer for non-code contributions like this? I am also happy to open a PR or schedule time to meet if that’s easiest.

Luckily, the questions are defined in YAML (or TOML!), so this can use a standard review process. I would also be happy to do a PEP/RFC-style review with questions written in a reST document, which might avoid the constraints of the question definition DSL.

---

Once again, thank you for both proposing the idea and for volunteering your firm's time to take forward this effort. I am very grateful, and aware that opportunities like this are few and far between.

Thanks,
Adam

[Carreau]

I would also be happy to do a PEP/RFC-style review with questions written in a reST document, which might avoid the constraints of the question definition DSL.

We used to do review/question edition in google doc or similar, the problem is that the document is never exactly what you just send to the survey software, so you always end up with translation issue problem between the discussions on the document between the contributor and the person updating the form.

The advantage of the DSL is that the changes/suggestions are non-ambiguous, and one can run the script that publish the survey with modification in order to see effects.

Some example of that, are:

• ranking vs rating,
• do you want "other" to just be "other", or leave the user the ability to enter their preferred value(s)
• some words in description put in bold, but form software does not support bold.

But yes initially question can be in a more free form document

[trallard]

I have no opposition in principle to a survey, though I don't personally have the time to organise the administration, beyond approving the questions.

I realise that perhaps our expectations were not entirely clear:

• Quansight Labs will take care of the logistics of running the survey, synthesis, and results-sharing
• What we are requesting from you (as in collective) is:
  • feedback on the survey and
  • support in sharing the survey with your users/community (if you want to be involved in the logistics from the above bullet, also let us know though)

@AA-Turner seems a number of the bullets you listed as things you'd like to know are covered in the draft survey we shared above https://github.com/Quansight-Labs/typeform-yaml-survey/blob/main/sphinx-survey.yml (yay!)
@isabela-pf is a UX designer at Quansight Labs and has a lot of experience in this type of work so we have someone available to bring in this expertise

I think such a survey should also make efforts to target both open-source (where we have a lot of visibility) and use in industry (where I'm aware Sphinx and competitors are used, but we have far less insight).

I agree. One of the goals of running this as a collaborative community effort is to reach a broader and more diverse group of users. This will also help us ensure this effort is valuable and helpful to the wider ecosystem (Sphinx maintainers/project, theme maintainers, infrastructure folks like RTD, etc.).

As much as this information is nice to know, though, it would be good to have agreed goals for what information we're seeking, and what decisions this will inform.

I agree here, too. I will come back later and add some thoughts here.

[AA-Turner]

Thank you Tania! I'd be happy to work with you on the analysis part, thank you for offering to run the survey logistics!

[isabela-pf]

Thank you all for taking the time to reply! From what I'm hearing, it seems like there's openness to review what we have and move forward.

As much as this information is nice to know, though, it would be good to have agreed goals for what information we're seeking, and what decisions this will inform.

I also entirely agree. So wonderful to hear this.

I made the current draft questions based on the following. Let me know if this would be easier to give feedback on another way.

• How are people using Sphinx? What are they using it for? The goal is to gain insight about and check our assumptions about Sphinx in the wild.
• Who are the people using Sphinx? The goal is to explore who is in the Sphinx community whether they are typically vocal about it or not.
• What are the strengths and weaknesses of Sphinx for people using it? What matters most to people? The goal is to be able to augment discussions on features and proposals to change them with feedback that can be followed up on.

[hugovk]

Here's some feedback on the questions at https://github.com/Quansight-Labs/typeform-yaml-survey/blob/main/sphinx-survey.yml

  - title: 'What documentation engines do you use?'
    multiple: true
    other: true
    choices:
      - 'MyST'
      - 'Docusaurus'
      - 'Mkdocs'
      - 'JupyterBook'
      - 'Sphinx'
      - 'None'
      - 'I don’t know'

Do things like Hugo, Astro, Pelican, Nikola, Eleventy etc count as doc engines? perhaps add a few more to the list? See also https://jamstack.org/generators/ and https://jamstack.org/headless-cms/

 - title: 'Where do you deploy your documentation pages?'
   multiple: true
   other: true
   choices:
     - 'GitHub Pages'
     - 'Netlify'
     - 'Read The Docs'
     - 'My institution has its own deployment'

Some others: Vercel, Cloudflare Pages, DigitalOcean, Hetzner, Render, Vultr.

  - title: 'What is the main reason you use Sphinx?'
    other: true
    choices:
      - 'Python API documentation generation (docstrings)'
      - 'Narrative documentation'
      - 'Educational materials'
      - 'Other projects in my ecosystem were using Sphinx'
      - 'Someone on my team knew how to use Sphinx'
      - 'I joined a project that was already using Sphinx'
      - 'Sphinx’s ease of use'
      - 'Sphinx’s performance'
      - 'Sphinx’s stability'
      - 'Sphinx’s release schedule'

For me, an important feature of Sphinx is being able to reference to other pages/sections, even if they move. And also across Sphinx sites with intersphinx.

  - title: 'What theme(s) do you use? Select all that apply.'
    multiple: true
    other: true
    choices:
    - 'Whatever comes with Sphinx by default'
    - 'Alabaster'
    - 'Read The Docs'
    - 'Pydata Sphinx Theme'
    - 'Material design'
    - 'Something else/I don’t know'

Let's also add Furo and maybe Python Docs Sphinx Theme.

[isabela-pf]

I can make all those changes! Is there a place related to Sphinx that would be good to open a PR so that people can review more directly? Or is it preferred to keep it at Quansight-Labs/typeform-yaml-survey for now?

[AA-Turner]

I'll create a new repository tomorrow.

[AA-Turner]

I have created https://github.com/sphinx-doc/meta and imported the current version of sphinx-survey.yml from the Quantsight repo. (That repo has no licence, so I have added "Copyright Quantsight" to the header of the file, hopefully this is acceptable.)

Hopefully this works for you @isabela-pf?

A

[isabela-pf]

Yes! Thank you so much @AA-Turner. I'll push changes there.

[AA-Turner]

For readers here, a PR has been opened with updates to the proposed survey: sphinx-doc/meta#2

[timhoffm]

Themes

I recently scaped the GitHub stars for the themes listed at https://sphinx-themes.org/. This should be a good first guess for the choices. The top 10 are:

• Read the Docs: 4666
• Furo: 2474
• Alabaster: 722
• Bootstrap: 588
• PyData Sphinx Theme: 557
• Sphinx Book Theme: 405
• Material: 306
• Press: 118
• Awesome: 99
• Piccolo: 87

I suggest to include everything up to including Material

Reason for using Sphinx

  - title: 'What is the main reason you use Sphinx?'
    other: true
    choices:
      - 'Python API documentation generation (docstrings)'
      - 'Narrative documentation'
      - 'Educational materials'
      - 'Other projects in my ecosystem were using Sphinx'
      - 'Someone on my team knew how to use Sphinx'
      - 'I joined a project that was already using Sphinx'
      - 'Sphinx’s ease of use'
      - 'Sphinx’s performance'
      - 'Sphinx’s stability'
      - 'Sphinx’s release schedule'

For the first three, I'm not clear whether they fit in or what they mean. The feel more like "What are you using spinx for?", which should be a separate question. The API documentation generation can additionally stay here but I suggest to rephrase as "API documentation generation from docstrings and links to that API docs" or similar.

I would probably leave out the last 3: All of them are not primary concerns. They have to be good enough to make a project usable, but they are not the key criterion whether to choose a project.

I'm missing:

• Add functionality through extensions

docutils

  - title: 'How comfortable do you feel when working with docutils?'

I don't understand the question. docutils is an underlying technology of Sphinx. For an end user all that is exposed is the ReST language defined by docutils. Is that what we're aiming at? Otherwise docutils is IMHO an implementation detail and only relevant for Sphinx and extension developers.

sphinx internals

  - title: 'What is the biggest challenge you face when searching for documentation on Sphinx internals?'

This is surprisingly specific. A regular user doesn't search for Sphinx internals. By definition they should not have to care about internals. Is this targeting sphinx developers, possible contributors, extension developers? Why do we specifically care are?

Other

Stakeholder type

Should we ask in the self-characterization for user / contributor / extension author / theme author? While the groups other than users are comparably small, I anticipate that they will take part overproportionally in the survey because the are more involved with the project, and they may have significantly different perspectives and answers compared to users. It therefore may be helpful to be able to filter on this when evaluating the survey. --> Quansight-Labs/typeform-yaml-survey#4

Rating

Does typeform support likert or rating scales?

I'd like to have insights into preference (How important is ...) and satisfaction (How satisfied are you with ...) on topics like

• Performance
• Stability
• Ease of use (Maybe subtopics: setup, learn, write, build, customize)
• Sphinx own documentation
• Default theme
• Builtin themes
• ReST vs. Markdown
• Extension ecosystem
• API documentation generation
• ...

[trallard]

For the first three, I'm not clear whether they fit in or what they mean. The feel more like "What are you using spinx for?", which should be a separate question. The API documentation generation can additionally stay here but I suggest to rephrase as "API documentation generation from docstrings and links to that API docs" or similar.

These questions ended up being merged somehow as we originally had a "what are you using Sphinx for" and "What is the main reason...." @isabela-pf let's check what happened here

[trallard]

I don't understand the question. docutils is an underlying technology of Sphinx. For an end user all that is exposed is the ReST language defined by docutils. Is that what we're aiming at? Otherwise docutils is IMHO an implementation detail and only relevant for Sphinx and extension developers.

Since we are targetting the community at large (including end-users and extension developers, among others), it is the same as the question regarding Sphinx internals. Both questions are related to contributor/developer experience.

Does typeform support likert or rating scales?

yes, we already have ranking in a couple (?) of questions (one?) for this precise reason, I do not see why we could not add more ranking questions if needed

[Carreau]

Feel free to send PRs against the Yaml file,

I'll see if I can integrate some of those changes.

My fear with too many options is they start to go out of the screen, and will discourage responders.

Does typeform support likert or rating scales?

IIRC it should have:

• rating (0-10)
• matrix rating (multiple 0-10 on a single screen)
• ranking (order for best to worst)

I just have not implemented the yaml-to-typeform API for the rating ones.

I also have only implemented basic yes/no jumps if we want jumps, but having jumps make designing surveys quite harder.

[isabela-pf]

I'm seconding this. I am aware of the same Typeform capabilities.

[AA-Turner]

I've noted the existence of the typeform-yaml-survey repo in https://github.com/sphinx-doc/meta, as I assume you intend to keep maintaining that in the pre-existing repo rather than moving it to Sphinx (happy either way).

[trallard]

It definitely makes sense to have this separate.